* Re: [for-next 4/6] net/mlx5: FPGA, Add basic support for Innova
From: Alexei Starovoitov @ 2017-06-06 18:34 UTC (permalink / raw)
To: David Miller
Cc: jgunthorpe, ilant, saeedm, dledford, netdev, linux-rdma,
jsorensen, andy.shevchenko, linux-fpga, atull, yi1.li, borisp
In-Reply-To: <20170606.134726.416045938000214127.davem@davemloft.net>
On Tue, Jun 06, 2017 at 01:47:26PM -0400, David Miller wrote:
> From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
> Date: Tue, 6 Jun 2017 10:42:35 -0700
>
> > so it's like rdma, but without using kernel rdma stack?
>
> No sockets here, just transformation rules. It's like offloading
> a complex TC rule to hardware version of that transformation.
>
> Yes, there is state, but I argue that it is no different than TC
> offloading rules. What if TC had "hash" and "crypt" operations
> and we attached them to appropriate u32 matches? You wouldn't
> be able to tell the difference.
there is huge difference in underlying hw.
fpga is a separate device with its own phy and mac layers, its
own queues, packet parsing and rdma logic.
Where as tc offload is happening within the same hw queues/memory/stats
management logic. My understanding that when I do 'ethtool -L' to
change number of queues or 'ethtool -G' it changes the memory layout
that tc offload is operating on as well.
When I do 'ethtool -S' it shows me the stats for the device
that tc offload rules are integral part of.
Whereas fpga is a different physical device with its own
buffers and such. We can add 'ethtool -G_fpga, -L_fpga', etc
but this type of discussion needs to happen _before_ the whole
thing is merged. It will never happen after the fact.
Just look at mlx responses, they still don't acknowledge the issue
and instead pushing for ipsec, tls (in other words: new features)
instead of addressing production issues that are obviously
not glamorous to work on and fix.
> I think you are way over-obsessed with this FPGA offload thing,
> quite frankly.
if we didn't have issues with eswitch that drops packets and
we don't even know how many, I wouldn't be complaining.
There is a discussion going on to add few counters for
eswitch visibility, but it's taking forever and it's not at
the point of exposing eswitch as a kernel object.
Why? because it's hard to refactor it now into something like devlink
or whatever new abstraction that would be needed.
^ permalink raw reply
* Re: [PATCH net-next 5/5] net: dsa: Stop accessing ds->dst->cpu_dp in drivers
From: Vivien Didelot @ 2017-06-06 18:33 UTC (permalink / raw)
To: Florian Fainelli, netdev; +Cc: andrew, john, davem
In-Reply-To: <99c1a096-2128-f259-0537-0456a117280c@gmail.com>
Florian Fainelli <f.fainelli@gmail.com> writes:
> On 06/06/2017 11:09 AM, Vivien Didelot wrote:
>> Florian Fainelli <f.fainelli@gmail.com> writes:
>>
>>> - phy_mode = of_get_phy_mode(ds->dst->cpu_dp->dn);
>>> + phy_mode = of_get_phy_mode(ds->ports[QCA8K_CPU_PORT].dn);
>>
>> Is it necessary to use QCA8K_CPU_PORT?
>>
>>> +static inline struct dsa_port *dsa_ds_get_cpu_dp(struct dsa_switch *ds)
>>> +{
>>> + return &ds->ports[fls(ds->cpu_port_mask) - 1];
>>> +}
>>
>> Wouldn't it be better to return the CPU port for a given port?
>> Something like return ds->ports[port].cpu_dp, so that we ease the
>> introduction of multiple CPU port a bit more?
>
> ds->ports[port].cpu_dp only gets assigned at dsa_slave_create() time,
> which is after ops->setup() has been called, hence this helper function
> in case you need it earlier (e.g: like mv88e6060).
I see no reason why we cannot assign ds->ports[x].cpu_dp before calling
ops->setup().
Even though it can be changed later, the DSA core can assign a dedicated
CPU port to each ports and only after that, call into drivers ops.
To me the DSA topology (dsa_ports, CPU, etc.) should be determined
before calling into the hardware and exposing slave interfaces to the
user. Otherwise it is confusing. Am I wrong?
Thanks,
Vivien
^ permalink raw reply
* Re: [PATCH v2 4/4] net: macb: Add hardware PTP support
From: Richard Cochran @ 2017-06-06 18:36 UTC (permalink / raw)
To: Rafal Ozieblo
Cc: David Miller,
nicolas.ferre-AIFe0yeh4nAAvxtiuMwx3w@public.gmane.org,
netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
devicetree-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
harini.katakam-gjFFaj9aHVfQT0dZR+AlfA@public.gmane.org,
andrei.pistirica-UWL1GkI3JZL3oGB3hsPCZA@public.gmane.org
In-Reply-To: <BN3PR07MB251673623B941CAF5DEAB474C9CB0-EldUQEzkDQfxGZiqM5fOI+FPX92sqiQdvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>
On Tue, Jun 06, 2017 at 08:54:50AM +0000, Rafal Ozieblo wrote:
> Would "ENOTSUP" be sufficient ?
You mean EOPNOTSUPP? Yes, sounds ok to me. EFAULT is surely wrong.
Thanks,
Richard
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [for-next 4/6] net/mlx5: FPGA, Add basic support for Innova
From: David Miller @ 2017-06-06 18:38 UTC (permalink / raw)
To: alexei.starovoitov
Cc: jgunthorpe, ilant, saeedm, dledford, netdev, linux-rdma,
jsorensen, andy.shevchenko, linux-fpga, atull, yi1.li, borisp
In-Reply-To: <20170606183457.fg7vgv2zji3fhlb5@ast-mbp.dhcp.thefacebook.com>
From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Date: Tue, 6 Jun 2017 11:34:59 -0700
> fpga is a separate device with its own phy and mac layers, its
> own queues, packet parsing and rdma logic.
Because that's how they bolted it onto the ASIC in current
implementation, it might not always be that way and be fully
integrated in the future.
And I stress the word "implementation" as in "implementation detail"
the visible behavior is going to be the same, the difference is how
the thing is hooked up and maybe how you program it.
^ permalink raw reply
* Re: [PATCH v2 3/4] net: macb: macb.c changed to macb_main.c
From: Richard Cochran @ 2017-06-06 18:39 UTC (permalink / raw)
To: Rafal Ozieblo
Cc: devicetree, netdev, nicolas.ferre, linux-kernel, andrei.pistirica,
harini.katakam, David Miller, linux-arm-kernel
In-Reply-To: <1496413661-21452-1-git-send-email-rafalo@cadence.com>
On Fri, Jun 02, 2017 at 03:27:41PM +0100, Rafal Ozieblo wrote:
> drivers/net/ethernet/cadence/macb.c | 3568 ------------------------------
> drivers/net/ethernet/cadence/macb_main.c | 3568 ++++++++++++++++++++++++++++++
You deleted macb.c, and so ...
> diff --git a/drivers/net/ethernet/cadence/Makefile b/drivers/net/ethernet/cadence/Makefile
> index 4ba7559..31ea6e3 100644
> --- a/drivers/net/ethernet/cadence/Makefile
> +++ b/drivers/net/ethernet/cadence/Makefile
> @@ -1,6 +1,7 @@
> #
> # Makefile for the Atmel network device drivers.
> #
> +macb-y := macb_main.o
>
> obj-$(CONFIG_MACB) += macb.o
how does this rule make sense any more?
Thanks,
Richard
^ permalink raw reply
* Re: More BPF verifier questions
From: Edward Cree @ 2017-06-06 18:48 UTC (permalink / raw)
To: Y Song
Cc: Alexei Starovoitov, Alexei Starovoitov, Daniel Borkmann, netdev,
David Miller, iovisor-dev
In-Reply-To: <CAH3MdRUasw4PL-Fq3tEbO=ByKGdCh4vLSF7EYZxtaDY0O24-Pg@mail.gmail.com>
On 05/06/17 08:06, Y Song wrote:
> On Fri, Jun 2, 2017 at 7:42 AM, Edward Cree <ecree@solarflare.com> wrote:
>> Test "helper access to variable memory: stack, bitwise AND + JMP, correct
>> bounds" is listed as expected to pass, but it passes zero in the 'size'
>> argument, an ARG_CONST_SIZE, to bpf_probe_read; I believe this should fail
>> (and with my WIP patch it does).
> Probably a typo or mis-statement. "size" is not passed in with "zero", but
> with an unknown value. Hence, it probably should fail.
>
> BPF_MOV64_IMM(BPF_REG_2, 16),
> BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_2, -128),
> BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, -128),
> BPF_ALU64_IMM(BPF_AND, BPF_REG_2, 64),
> BPF_MOV64_IMM(BPF_REG_4, 0),
> BPF_JMP_REG(BPF_JGE, BPF_REG_4, BPF_REG_2, 2),
> BPF_MOV64_IMM(BPF_REG_3, 0),
> BPF_EMIT_CALL(BPF_FUNC_probe_read),
So, in fact this unknown value is really 16 & 64 == 0, but the verifier doesn't
know that and concludes that it's either 0 or 64 (after the AND). But then
what I didn't spot before, and now have, is that the BPF_JGE tests if 0 >= size.
Since we're in the false branch, that means size > 0, and so we're fine.
The test case is correct, and now that I've fixed the min/max tracking in my
patches, the verifier accepts it again.
-Ed
^ permalink raw reply
* Re: [for-next 4/6] net/mlx5: FPGA, Add basic support for Innova
From: Alexei Starovoitov @ 2017-06-06 18:55 UTC (permalink / raw)
To: David Miller
Cc: jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/,
ilant-VPRAkNaXOzVWk0Htik3J/w,
saeedm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb,
dledford-H+wXaHxf7aLQT0dZR+AlfA, netdev-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, jsorensen-b10kYP2dOMg,
andy.shevchenko-Re5JQEeQqe8AvxtiuMwx3w,
linux-fpga-u79uwXL29TY76Z2rM5mHXA, yi1.li-VuQAYsv1563Yd54FQh9/CA,
borisp-VPRAkNaXOzVWk0Htik3J/w
In-Reply-To: <20170606.143824.717466091308335341.davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
On Tue, Jun 06, 2017 at 02:38:24PM -0400, David Miller wrote:
> From: Alexei Starovoitov <alexei.starovoitov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> Date: Tue, 6 Jun 2017 11:34:59 -0700
>
> > fpga is a separate device with its own phy and mac layers, its
> > own queues, packet parsing and rdma logic.
>
> Because that's how they bolted it onto the ASIC in current
> implementation, it might not always be that way and be fully
> integrated in the future.
>
> And I stress the word "implementation" as in "implementation detail"
> the visible behavior is going to be the same, the difference is how
> the thing is hooked up and maybe how you program it.
whether fpga is a separate chip or part of the same asic makes no difference.
They are still different devices from sw point of view.
If in the future mlx will make it into the nic in a way that encryption shares
all memory management logic and there is no fpga at all then it indeed will
be similar to tc offload. Right now it's not and needs different sw architecture.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 3/4] net: macb: macb.c changed to macb_main.c
From: David Miller @ 2017-06-06 19:00 UTC (permalink / raw)
To: richardcochran-Re5JQEeQqe8AvxtiuMwx3w
Cc: rafalo-vna1KIf7WgpBDgjK7y7TUQ,
nicolas.ferre-AIFe0yeh4nAAvxtiuMwx3w,
netdev-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
harini.katakam-gjFFaj9aHVfQT0dZR+AlfA,
andrei.pistirica-UWL1GkI3JZL3oGB3hsPCZA
In-Reply-To: <20170606183933.GA1205-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
From: Richard Cochran <richardcochran-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date: Tue, 6 Jun 2017 20:39:33 +0200
> On Fri, Jun 02, 2017 at 03:27:41PM +0100, Rafal Ozieblo wrote:
>> drivers/net/ethernet/cadence/macb.c | 3568 ------------------------------
>> drivers/net/ethernet/cadence/macb_main.c | 3568 ++++++++++++++++++++++++++++++
>
> You deleted macb.c, and so ...
Rename macb.c to macb_main.c
>> +macb-y := macb_main.o
>>
>> obj-$(CONFIG_MACB) += macb.o
>
> how does this rule make sense any more?
He's adjusting the Makefile so that it build macb_main.c into macb.o
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [for-next 4/6] net/mlx5: FPGA, Add basic support for Innova
From: David Miller @ 2017-06-06 19:01 UTC (permalink / raw)
To: alexei.starovoitov-Re5JQEeQqe8AvxtiuMwx3w
Cc: jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/,
ilant-VPRAkNaXOzVWk0Htik3J/w,
saeedm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb,
dledford-H+wXaHxf7aLQT0dZR+AlfA, netdev-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, jsorensen-b10kYP2dOMg,
andy.shevchenko-Re5JQEeQqe8AvxtiuMwx3w,
linux-fpga-u79uwXL29TY76Z2rM5mHXA, yi1.li-VuQAYsv1563Yd54FQh9/CA,
borisp-VPRAkNaXOzVWk0Htik3J/w
In-Reply-To: <20170606185532.2byjdonwsyan2asl-+o4/htvd0TCa6kscz5V53/3mLCh9rsb+VpNB7YpNyf8@public.gmane.org>
From: Alexei Starovoitov <alexei.starovoitov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date: Tue, 6 Jun 2017 11:55:33 -0700
> If in the future mlx will make it into the nic in a way that
> encryption shares all memory management logic and there is no fpga
> at all then it indeed will be similar to tc offload. Right now it's
> not and needs different sw architecture.
If the visible effect is identical, I fundamentally disagree with you.
I don't care if there is a frog sitting on the PHY that transforms
the packets, it's all the same if the visible behavior is identical.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v1] net: phy: Delete unused function phy_ethtool_gset
From: David Miller @ 2017-06-06 19:12 UTC (permalink / raw)
To: yuval.shaia; +Cc: f.fainelli, andrew, corbet, netdev, linux-doc
In-Reply-To: <20170605071840.16492-1-yuval.shaia@oracle.com>
From: Yuval Shaia <yuval.shaia@oracle.com>
Date: Mon, 5 Jun 2017 10:18:40 +0300
> It's unused, so remove it.
>
> Signed-off-by: Yuval Shaia <yuval.shaia@oracle.com>
> ---
> v0 -> v1:
> * Add commit message
> * Update Documentation/networking/phy.txt
> * Modify commit header message
Applied to net-next, thanks.
^ permalink raw reply
* Re: [PATCH][V2] net: stmmac: ensure jumbo_frm error return is correctly checked for -ve value
From: David Miller @ 2017-06-06 19:14 UTC (permalink / raw)
To: colin.king
Cc: peppe.cavallaro, alexandre.torgue, netdev, kernel-janitors,
linux-kernel
In-Reply-To: <20170605090452.30688-1-colin.king@canonical.com>
From: Colin King <colin.king@canonical.com>
Date: Mon, 5 Jun 2017 10:04:52 +0100
> From: Colin Ian King <colin.king@canonical.com>
>
> The current comparison of entry < 0 will never be true since entry is an
> unsigned integer. Make entry an int to ensure -ve error return values
> from the call to jumbo_frm are correctly being caught.
>
> Detected by CoverityScan, CID#1238760 ("Macro compares unsigned to 0")
>
> Signed-off-by: Colin Ian King <colin.king@canonical.com>
I know Julia asked for more comments, but I'm going to apply this
as-is for now as it is correct.
Thanks Colin.
^ permalink raw reply
* Fw: [Bug 195969] New: ipsec icmp and udp works, tcp doesn't work
From: Stephen Hemminger @ 2017-06-06 19:14 UTC (permalink / raw)
To: Steffen Klassert, Herbert Xu; +Cc: netdev
Begin forwarded message:
Date: Sat, 03 Jun 2017 06:25:05 +0000
From: bugzilla-daemon@bugzilla.kernel.org
To: stephen@networkplumber.org
Subject: [Bug 195969] New: ipsec icmp and udp works, tcp doesn't work
https://bugzilla.kernel.org/show_bug.cgi?id=195969
Bug ID: 195969
Summary: ipsec icmp and udp works, tcp doesn't work
Product: Networking
Version: 2.5
Kernel Version: 4.11.3-1-ARCH
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: stephen@networkplumber.org
Reporter: dev@djagoo.io
Regression: No
A few days ago I updated to 4.11.3-1-ARCH. After that my VPN access to our
corporate network was broken.
The connection is established and I can use UDP (i.e. DNS) and ICMP. All TCP
connections I tried (ssh, smb, http...) failed.
On the AUR page "MartinDiehl commented on 2017-05-25 19:57" the same error.
https://aur.archlinux.org/packages/strongswan/
And I found a bug report on redhat bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=1458222
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply
* Re: [PATCH] ppp: mppe: Use vsnprintf extension %phN
From: David Miller @ 2017-06-06 19:16 UTC (permalink / raw)
To: joe; +Cc: paulus, linux-ppp, netdev, linux-kernel
In-Reply-To: <80f51106926c82374f6555b54af71a674b8bf94e.1496665332.git.joe@perches.com>
From: Joe Perches <joe@perches.com>
Date: Mon, 5 Jun 2017 05:22:50 -0700
> Using this extension reduces the object size.
>
> $ size drivers/net/ppp/ppp_mppe.o*
> text data bss dec hex filename
> 5683 216 8 5907 1713 drivers/net/ppp/ppp_mppe.o.new
> 5808 216 8 6032 1790 drivers/net/ppp/ppp_mppe.o.old
>
> Signed-off-by: Joe Perches <joe@perches.com>
Applied to net-next, thanks Joe.
^ permalink raw reply
* Re: [PATCH] cxgb4: implement ndo_set_vf_rate()
From: David Miller @ 2017-06-06 19:17 UTC (permalink / raw)
To: ganeshgr; +Cc: netdev, nirranjan, kumaras
In-Reply-To: <1496667860-11703-1-git-send-email-ganeshgr@chelsio.com>
From: Ganesh Goudar <ganeshgr@chelsio.com>
Date: Mon, 5 Jun 2017 18:34:20 +0530
> Implement ndo_set_vf_rate() for mgmt interface to support rate-limiting
> of VF traffic using 'ip' command.
>
> Based on the original work of Kumar Sanghvi <kumaras@chelsio.com>
>
> Signed-off-by: Ganesh Goudar <ganeshgr@chelsio.com>
Applied to net-next, thanks Ganesh.
^ permalink raw reply
* Re: [PATCH] net/ipv6: Fix CALIPSO causing GPF with datagram support
From: David Miller @ 2017-06-06 19:19 UTC (permalink / raw)
To: richard_c_haines; +Cc: paul, netdev, selinux, linux-security-module
In-Reply-To: <20170605154440.2434-1-richard_c_haines@btinternet.com>
From: Richard Haines <richard_c_haines@btinternet.com>
Date: Mon, 5 Jun 2017 16:44:40 +0100
> When using CALIPSO with IPPROTO_UDP it is possible to trigger a GPF as the
> IP header may have moved.
>
> Also update the payload length after adding the CALIPSO option.
>
> Signed-off-by: Richard Haines <richard_c_haines@btinternet.com>
Applied and queued up for -stable, thank you Richard.
^ permalink raw reply
* [PATCH net-next] tcp: add TCPMemoryPressuresChrono counter
From: Eric Dumazet @ 2017-06-06 19:19 UTC (permalink / raw)
To: David Miller; +Cc: netdev
From: Eric Dumazet <edumazet@google.com>
DRAM supply shortage and poor memory pressure tracking in TCP
stack makes any change in SO_SNDBUF/SO_RCVBUF (or equivalent autotuning
limits) and tcp_mem[] quite hazardous.
TCPMemoryPressures SNMP counter is an indication of tcp_mem sysctl
limits being hit, but only tracking number of transitions.
If TCP stack behavior under stress was perfect :
1) It would maintain memory usage close to the limit.
2) Memory pressure state would be entered for short times.
We certainly prefer 100 events lasting 10ms compared to one event
lasting 200 seconds.
This patch adds a new SNMP counter tracking cumulative duration of
memory pressure events, given in ms units.
$ cat /proc/sys/net/ipv4/tcp_mem
3088 4117 6176
$ grep TCP /proc/net/sockstat
TCP: inuse 180 orphan 0 tw 2 alloc 234 mem 4140
$ nstat -n ; sleep 10 ; nstat |grep Pressure
TcpExtTCPMemoryPressures 1700
TcpExtTCPMemoryPressuresChrono 5209
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
include/net/sock.h | 22 ++--------------------
include/net/tcp.h | 3 ++-
include/uapi/linux/snmp.h | 1 +
net/core/sock.c | 20 ++++++++++++++++++++
net/decnet/af_decnet.c | 2 +-
net/ipv4/proc.c | 1 +
net/ipv4/tcp.c | 27 +++++++++++++++++++++++----
net/ipv4/tcp_ipv4.c | 1 +
net/ipv6/tcp_ipv6.c | 1 +
net/sctp/socket.c | 2 +-
10 files changed, 53 insertions(+), 27 deletions(-)
diff --git a/include/net/sock.h b/include/net/sock.h
index 3467d9e89e7dba1c35fa44a6268a28735f795319..858891c36f94ad2577726d6d21cf871dbcd55d98 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1080,6 +1080,7 @@ struct proto {
bool (*stream_memory_free)(const struct sock *sk);
/* Memory pressure */
void (*enter_memory_pressure)(struct sock *sk);
+ void (*leave_memory_pressure)(struct sock *sk);
atomic_long_t *memory_allocated; /* Current allocated memory. */
struct percpu_counter *sockets_allocated; /* Current number of sockets. */
/*
@@ -1088,7 +1089,7 @@ struct proto {
* All the __sk_mem_schedule() is of this nature: accounting
* is strict, actions are advisory and have some latency.
*/
- int *memory_pressure;
+ unsigned long *memory_pressure;
long *sysctl_mem;
int *sysctl_wmem;
int *sysctl_rmem;
@@ -1193,25 +1194,6 @@ static inline bool sk_under_memory_pressure(const struct sock *sk)
return !!*sk->sk_prot->memory_pressure;
}
-static inline void sk_leave_memory_pressure(struct sock *sk)
-{
- int *memory_pressure = sk->sk_prot->memory_pressure;
-
- if (!memory_pressure)
- return;
-
- if (*memory_pressure)
- *memory_pressure = 0;
-}
-
-static inline void sk_enter_memory_pressure(struct sock *sk)
-{
- if (!sk->sk_prot->enter_memory_pressure)
- return;
-
- sk->sk_prot->enter_memory_pressure(sk);
-}
-
static inline long
sk_memory_allocated(const struct sock *sk)
{
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 82462db97183abebb33628eb5e04a5c5f04ea873..03482a5a069a18c776bd2071f0d74c8e56c0bed2 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -279,7 +279,7 @@ extern int sysctl_tcp_pacing_ca_ratio;
extern atomic_long_t tcp_memory_allocated;
extern struct percpu_counter tcp_sockets_allocated;
-extern int tcp_memory_pressure;
+extern unsigned long tcp_memory_pressure;
/* optimized version of sk_under_memory_pressure() for TCP sockets */
static inline bool tcp_under_memory_pressure(const struct sock *sk)
@@ -1322,6 +1322,7 @@ extern void tcp_openreq_init_rwin(struct request_sock *req,
const struct dst_entry *dst);
void tcp_enter_memory_pressure(struct sock *sk);
+void tcp_leave_memory_pressure(struct sock *sk);
static inline int keepalive_intvl_when(const struct tcp_sock *tp)
{
diff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h
index 95cffcb21dfdba7c974706131d0f43e21435e82d..d8569329579816213255169d0c183f4400835f7b 100644
--- a/include/uapi/linux/snmp.h
+++ b/include/uapi/linux/snmp.h
@@ -228,6 +228,7 @@ enum
LINUX_MIB_TCPABORTONLINGER, /* TCPAbortOnLinger */
LINUX_MIB_TCPABORTFAILED, /* TCPAbortFailed */
LINUX_MIB_TCPMEMORYPRESSURES, /* TCPMemoryPressures */
+ LINUX_MIB_TCPMEMORYPRESSURESCHRONO, /* TCPMemoryPressuresChrono */
LINUX_MIB_TCPSACKDISCARD, /* TCPSACKDiscard */
LINUX_MIB_TCPDSACKIGNOREDOLD, /* TCPSACKIgnoredOld */
LINUX_MIB_TCPDSACKIGNOREDNOUNDO, /* TCPSACKIgnoredNoUndo */
diff --git a/net/core/sock.c b/net/core/sock.c
index bef844127e0182091678b9d57f7ec85c5241748d..ad8a4bc841267a442a1da3c56ef1cf074f9825b9 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2076,6 +2076,26 @@ int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
}
EXPORT_SYMBOL(sock_cmsg_send);
+static void sk_enter_memory_pressure(struct sock *sk)
+{
+ if (!sk->sk_prot->enter_memory_pressure)
+ return;
+
+ sk->sk_prot->enter_memory_pressure(sk);
+}
+
+static void sk_leave_memory_pressure(struct sock *sk)
+{
+ if (sk->sk_prot->leave_memory_pressure) {
+ sk->sk_prot->leave_memory_pressure(sk);
+ } else {
+ unsigned long *memory_pressure = sk->sk_prot->memory_pressure;
+
+ if (memory_pressure && *memory_pressure)
+ *memory_pressure = 0;
+ }
+}
+
/* On 32bit arches, an skb frag is limited to 2^15 */
#define SKB_FRAG_PAGE_ORDER get_order(32768)
diff --git a/net/decnet/af_decnet.c b/net/decnet/af_decnet.c
index 405483a07efc7ac2efcfe86e285a7673547c9691..73a0399dc7a277178b0a432a067172131dce99ee 100644
--- a/net/decnet/af_decnet.c
+++ b/net/decnet/af_decnet.c
@@ -447,7 +447,7 @@ static void dn_destruct(struct sock *sk)
dst_release(rcu_dereference_check(sk->sk_dst_cache, 1));
}
-static int dn_memory_pressure;
+static unsigned long dn_memory_pressure;
static void dn_enter_memory_pressure(struct sock *sk)
{
diff --git a/net/ipv4/proc.c b/net/ipv4/proc.c
index fa44e752a9a3f8eb9957314149ae15e6df10465a..43eb6567b3a0a2add9a1d36019eae5b6d5caf657 100644
--- a/net/ipv4/proc.c
+++ b/net/ipv4/proc.c
@@ -250,6 +250,7 @@ static const struct snmp_mib snmp4_net_list[] = {
SNMP_MIB_ITEM("TCPAbortOnLinger", LINUX_MIB_TCPABORTONLINGER),
SNMP_MIB_ITEM("TCPAbortFailed", LINUX_MIB_TCPABORTFAILED),
SNMP_MIB_ITEM("TCPMemoryPressures", LINUX_MIB_TCPMEMORYPRESSURES),
+ SNMP_MIB_ITEM("TCPMemoryPressuresChrono", LINUX_MIB_TCPMEMORYPRESSURESCHRONO),
SNMP_MIB_ITEM("TCPSACKDiscard", LINUX_MIB_TCPSACKDISCARD),
SNMP_MIB_ITEM("TCPDSACKIgnoredOld", LINUX_MIB_TCPDSACKIGNOREDOLD),
SNMP_MIB_ITEM("TCPDSACKIgnoredNoUndo", LINUX_MIB_TCPDSACKIGNOREDNOUNDO),
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index f7be94fc8431d30a6b8c5de62a70b529789f7873..00d1bf4092fdf852e54eccf8ec5b130ac9935704 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -320,18 +320,37 @@ struct tcp_splice_state {
* All the __sk_mem_schedule() is of this nature: accounting
* is strict, actions are advisory and have some latency.
*/
-int tcp_memory_pressure __read_mostly;
+unsigned long tcp_memory_pressure __read_mostly;
EXPORT_SYMBOL(tcp_memory_pressure);
void tcp_enter_memory_pressure(struct sock *sk)
{
- if (!tcp_memory_pressure) {
+ unsigned long val;
+
+ if (tcp_memory_pressure)
+ return;
+ val = jiffies;
+
+ if (!val)
+ val--;
+ if (!cmpxchg(&tcp_memory_pressure, 0, val))
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMEMORYPRESSURES);
- tcp_memory_pressure = 1;
- }
}
EXPORT_SYMBOL(tcp_enter_memory_pressure);
+void tcp_leave_memory_pressure(struct sock *sk)
+{
+ unsigned long val;
+
+ if (!tcp_memory_pressure)
+ return;
+ val = xchg(&tcp_memory_pressure, 0);
+ if (val)
+ NET_ADD_STATS(sock_net(sk), LINUX_MIB_TCPMEMORYPRESSURESCHRONO,
+ jiffies_to_msecs(jiffies - val));
+}
+EXPORT_SYMBOL(tcp_leave_memory_pressure);
+
/* Convert seconds to retransmits based on initial and max timeout */
static u8 secs_to_retrans(int seconds, int timeout, int rto_max)
{
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 191b2f78b19d2c8d62c59cc046bd608687679619..d19933949373ca019fd06fa310a506efd29718cb 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -2387,6 +2387,7 @@ struct proto tcp_prot = {
.unhash = inet_unhash,
.get_port = inet_csk_get_port,
.enter_memory_pressure = tcp_enter_memory_pressure,
+ .leave_memory_pressure = tcp_leave_memory_pressure,
.stream_memory_free = tcp_stream_memory_free,
.sockets_allocated = &tcp_sockets_allocated,
.orphan_count = &tcp_orphan_count,
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 233edfabe1dbceaeb6cdd42a2bb379072aeee361..f6592790d9f93bf40f3e5432c945bca20b28ff34 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1909,6 +1909,7 @@ struct proto tcpv6_prot = {
.unhash = inet_unhash,
.get_port = inet_csk_get_port,
.enter_memory_pressure = tcp_enter_memory_pressure,
+ .leave_memory_pressure = tcp_leave_memory_pressure,
.stream_memory_free = tcp_stream_memory_free,
.sockets_allocated = &tcp_sockets_allocated,
.memory_allocated = &tcp_memory_allocated,
diff --git a/net/sctp/socket.c b/net/sctp/socket.c
index 0822046e4f3f5a1acd3f5382d915bf9004a25c1c..5f58dd03e3ace38b9c4babbe2d92f0a3f98a4b68 100644
--- a/net/sctp/socket.c
+++ b/net/sctp/socket.c
@@ -103,7 +103,7 @@ static int sctp_autobind(struct sock *sk);
static void sctp_sock_migrate(struct sock *, struct sock *,
struct sctp_association *, sctp_socket_type_t);
-static int sctp_memory_pressure;
+static unsigned long sctp_memory_pressure;
static atomic_long_t sctp_memory_allocated;
struct percpu_counter sctp_sockets_allocated;
^ permalink raw reply related
* Re: [PATCH net-next 0/5] net: dsa: Multi-CPU ground work
From: Florian Fainelli @ 2017-06-06 19:20 UTC (permalink / raw)
To: Vivien Didelot, netdev; +Cc: andrew, john, davem
In-Reply-To: <87bmq1j7ni.fsf@weeman.i-did-not-set--mail-host-address--so-tickle-me>
On 06/06/2017 11:25 AM, Vivien Didelot wrote:
> Hi Florian,
>
> Florian Fainelli <f.fainelli@gmail.com> writes:
>
>> This patch series prepares the ground for adding mutliple CPU port support to
>
> multiple
>
>> DSA, and starts by removing redundant pieces of information such as
>> master_netdev which is cpu_dp->ethernet. Finally drivers are moved away from
>
> cpu_dp->netdev
>
>> directly accessing ds->dst->cpu_dp and use appropriate helper functions.
>>
>> Note that if you have Device Tree blobs/platform configurations that are
>> currently listing multiple CPU ports, the proposed behavior in
>> dsa_ds_get_cpu_dp() will be to return the last bit set in ds->cpu_port_mask.
>>
>> Future plans include:
>> - making dst->cpu_dp a flexible data structure (array, list, you name it)
>> - having the ability for drivers to return a default/preferred CPU port (if
>> necessary)
>
> The overall patchset looks good. I have questions for future work
> though.
>
> I am still not sure that we need CPU port references in
> dsa_switch_tree. When device tree or pdata is parsed, we have allocated
> dsa_switch and dsa_port structures. We should be able validate and
> assign all ds->ports[x].cpu_dp, before setting up the switches and
> creating the slave devices. What do you think?
True, we should be able to do that, thanks for the suggestion.
>
> Also I see dsa_ptr becoming a pointer to the assosicated dsa_port, and
> dsa_port should contain the tagging ops for quick access. That is more
> rigourous with the physical representation and much easier for
> transparent multi-CPU port support.
Ultimately, I agree we should probably have dev->dsa_ptr be the actual
CPU port within the switch, and from the switch be able to go to the
collection of switches (dst). This should indeed be a bit more optimized
as there should be less traversal of structures in such a case.
Thanks!
--
Florian
^ permalink raw reply
* Re: [PATCH v3 net-next 0/8] Introduce bpf ID
From: David Miller @ 2017-06-06 19:41 UTC (permalink / raw)
To: kafai; +Cc: netdev, ast, daniel, kernel-team
In-Reply-To: <cover.1496686606.git.kafai@fb.com>
From: Martin KaFai Lau <kafai@fb.com>
Date: Mon, 5 Jun 2017 12:15:45 -0700
> This patch series:
> 1) Introduce ID for both bpf_prog and bpf_map.
> 2) Add bpf commands to iterate the prog IDs and map
> IDs of the system.
> 3) Add bpf commands to get a prog/map fd from an ID
> 4) Add bpf command to get prog/map info from a fd.
> The prog/map info is a jump start in this patchset
> and it is not meant to be a complete list. They can
> be extended in the future patches.
Looks good, series applied, thanks Martin.
^ permalink raw reply
* Re: [PATCH v2] arm: eBPF JIT compiler
From: Shubham Bansal @ 2017-06-06 19:47 UTC (permalink / raw)
To: Kees Cook
Cc: Network Development, Daniel Borkmann, David S. Miller,
Alexei Starovoitov, Russell King,
linux-arm-kernel@lists.infradead.org, LKML, Andrew Lunn
In-Reply-To: <CAGXu5jKmLNMR+uhM9Ov2XPW6kCjip_SjScgEH0fTFp6fQtKFiA@mail.gmail.com>
Hi Russell, Alexei, David, Daniel, kees,
Any update on this patch moving forward?
Best,
Shubham Bansal
On Wed, May 31, 2017 at 12:49 AM, Kees Cook <keescook@chromium.org> wrote:
> Forwarding this to net-dev and eBPF folks, who weren't on CC...
>
> -Kees
>
> On Thu, May 25, 2017 at 4:13 PM, Shubham Bansal
> <illusionist.neo@gmail.com> wrote:
>> The JIT compiler emits ARM 32 bit instructions. Currently, It supports
>> eBPF only. Classic BPF is supported because of the conversion by BPF
>> core.
>>
>> This patch is essentially changing the current implementation of JIT
>> compiler of Berkeley Packet Filter from classic to internal with almost
>> all instructions from eBPF ISA supported except the following
>> BPF_ALU64 | BPF_DIV | BPF_K
>> BPF_ALU64 | BPF_DIV | BPF_X
>> BPF_ALU64 | BPF_MOD | BPF_K
>> BPF_ALU64 | BPF_MOD | BPF_X
>> BPF_STX | BPF_XADD | BPF_W
>> BPF_STX | BPF_XADD | BPF_DW
>> BPF_JMP | BPF_CALL
>>
>> Implementation is using scratch space to emulate 64 bit eBPF ISA on 32 bit
>> ARM because of deficiency of general purpose registers on ARM. Currently,
>> only LITTLE ENDIAN machines are supported in this eBPF JIT Compiler.
>>
>> Tested on ARMv7 with QEMU by me (Shubham Bansal).
>> Tested on ARMv5 by Andrew Lunn (andrew@lunn.ch).
>> Expected to work on ARMv6 as well, as its a part ARMv7 and part ARMv5.
>> Although, a proper testing is not done for ARMv6.
>>
>> Both of these testing are done with and without CONFIG_FRAME_POINTER
>> separately for LITTLE ENDIAN machine.
>>
>> For testing:
>>
>> 1. JIT is enabled with
>> echo 1 > /proc/sys/net/core/bpf_jit_enable
>> 2. Constant Blinding can be enabled along with JIT using
>> echo 1 > /proc/sys/net/core/bpf_jit_enable
>> echo 2 > /proc/sys/net/core/bpf_jit_harden
>>
>> See Documentation/networking/filter.txt for more information.
>>
>> Result : test_bpf: Summary: 314 PASSED, 0 FAILED, [278/306 JIT'ed]
>>
>> Signed-off-by: Shubham Bansal <illusionist.neo@gmail.com>
>> ---
>> Documentation/networking/filter.txt | 4 +-
>> arch/arm/Kconfig | 2 +-
>> arch/arm/net/bpf_jit_32.c | 2404 ++++++++++++++++++++++++-----------
>> arch/arm/net/bpf_jit_32.h | 108 +-
>> 4 files changed, 1713 insertions(+), 805 deletions(-)
>>
>> diff --git a/Documentation/networking/filter.txt b/Documentation/networking/filter.txt
>> index b69b205..01165ac 100644
>> --- a/Documentation/networking/filter.txt
>> +++ b/Documentation/networking/filter.txt
>> @@ -596,8 +596,8 @@ skb pointer). All constraints and restrictions from bpf_check_classic() apply
>> before a conversion to the new layout is being done behind the scenes!
>>
>> Currently, the classic BPF format is being used for JITing on most 32-bit
>> -architectures, whereas x86-64, aarch64, s390x, powerpc64, sparc64 perform JIT
>> -compilation from eBPF instruction set.
>> +architectures, whereas x86-64, aarch64, arm, s390x, powerpc64, sparc64 perform
>> +JIT compilation from eBPF instruction set.
>>
>> Some core changes of the new internal format:
>>
>> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
>> index 8a7ab5e..13ade46 100644
>> --- a/arch/arm/Kconfig
>> +++ b/arch/arm/Kconfig
>> @@ -47,7 +47,7 @@ config ARM
>> select HAVE_ARCH_SECCOMP_FILTER if (AEABI && !OABI_COMPAT)
>> select HAVE_ARCH_TRACEHOOK
>> select HAVE_ARM_SMCCC if CPU_V7
>> - select HAVE_CBPF_JIT
>> + select HAVE_EBPF_JIT
>> select HAVE_CC_STACKPROTECTOR
>> select HAVE_CONTEXT_TRACKING
>> select HAVE_C_RECORDMCOUNT
>> diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c
>> index 93d0b6d..c7476e5 100644
>> --- a/arch/arm/net/bpf_jit_32.c
>> +++ b/arch/arm/net/bpf_jit_32.c
>> @@ -1,13 +1,15 @@
>> /*
>> - * Just-In-Time compiler for BPF filters on 32bit ARM
>> + * Just-In-Time compiler for eBPF filters on 32bit ARM
>> *
>> * Copyright (c) 2011 Mircea Gherzan <mgherzan@gmail.com>
>> + * Copyright (c) 2017 Shubham Bansal <illusionist.neo@gmail.com>
>> *
>> * This program is free software; you can redistribute it and/or modify it
>> * under the terms of the GNU General Public License as published by the
>> * Free Software Foundation; version 2 of the License.
>> */
>>
>> +#include <linux/bpf.h>
>> #include <linux/bitops.h>
>> #include <linux/compiler.h>
>> #include <linux/errno.h>
>> @@ -23,44 +25,91 @@
>>
>> #include "bpf_jit_32.h"
>>
>> +int bpf_jit_enable __read_mostly;
>> +
>> +#define STACK_OFFSET(k) (k)
>> +#define TMP_REG_1 (MAX_BPF_JIT_REG + 0) /* TEMP Register 1 */
>> +#define TMP_REG_2 (MAX_BPF_JIT_REG + 1) /* TEMP Register 2 */
>> +#define TCALL_CNT (MAX_BPF_JIT_REG + 2) /* Tail Call Count */
>> +
>> +/* Flags used for JIT optimization */
>> +#define SEEN_CALL (1 << 0)
>> +
>> +#define FLAG_IMM_OVERFLOW (1 << 0)
>> +
>> /*
>> - * ABI:
>> + * Map eBPF registers to ARM 32bit registers or stack scratch space.
>> + *
>> + * 1. First argument is passed using the arm 32bit registers and rest of the
>> + * arguments are passed on stack scratch space.
>> + * 2. First callee-saved aregument is mapped to arm 32 bit registers and rest
>> + * arguments are mapped to scratch space on stack.
>> + * 3. We need two 64 bit temp registers to do complex operations on eBPF
>> + * registers.
>> + *
>> + * As the eBPF registers are all 64 bit registers and arm has only 32 bit
>> + * registers, we have to map each eBPF registers with two arm 32 bit regs or
>> + * scratch memory space and we have to build eBPF 64 bit register from those.
>> *
>> - * r0 scratch register
>> - * r4 BPF register A
>> - * r5 BPF register X
>> - * r6 pointer to the skb
>> - * r7 skb->data
>> - * r8 skb_headlen(skb)
>> */
>> +static const u8 bpf2a32[][2] = {
>> + /* return value from in-kernel function, and exit value from eBPF */
>> + [BPF_REG_0] = {ARM_R1, ARM_R0},
>> + /* arguments from eBPF program to in-kernel function */
>> + [BPF_REG_1] = {ARM_R3, ARM_R2},
>> + /* Stored on stack scratch space */
>> + [BPF_REG_2] = {STACK_OFFSET(0), STACK_OFFSET(4)},
>> + [BPF_REG_3] = {STACK_OFFSET(8), STACK_OFFSET(12)},
>> + [BPF_REG_4] = {STACK_OFFSET(16), STACK_OFFSET(20)},
>> + [BPF_REG_5] = {STACK_OFFSET(24), STACK_OFFSET(28)},
>> + /* callee saved registers that in-kernel function will preserve */
>> + [BPF_REG_6] = {ARM_R5, ARM_R4},
>> + /* Stored on stack scratch space */
>> + [BPF_REG_7] = {STACK_OFFSET(32), STACK_OFFSET(36)},
>> + [BPF_REG_8] = {STACK_OFFSET(40), STACK_OFFSET(44)},
>> + [BPF_REG_9] = {STACK_OFFSET(48), STACK_OFFSET(52)},
>> + /* Read only Frame Pointer to access Stack */
>> + [BPF_REG_FP] = {STACK_OFFSET(56), STACK_OFFSET(60)},
>> + /* Temporary Register for internal BPF JIT, can be used
>> + * for constant blindings and others.
>> + */
>> + [TMP_REG_1] = {ARM_R7, ARM_R6},
>> + [TMP_REG_2] = {ARM_R10, ARM_R8},
>> + /* Tail call count. Stored on stack scratch space. */
>> + [TCALL_CNT] = {STACK_OFFSET(64), STACK_OFFSET(68)},
>> + /* temporary register for blinding constants.
>> + * Stored on stack scratch space.
>> + */
>> + [BPF_REG_AX] = {STACK_OFFSET(72), STACK_OFFSET(76)},
>> +};
>>
>> -#define r_scratch ARM_R0
>> -/* r1-r3 are (also) used for the unaligned loads on the non-ARMv7 slowpath */
>> -#define r_off ARM_R1
>> -#define r_A ARM_R4
>> -#define r_X ARM_R5
>> -#define r_skb ARM_R6
>> -#define r_skb_data ARM_R7
>> -#define r_skb_hl ARM_R8
>> -
>> -#define SCRATCH_SP_OFFSET 0
>> -#define SCRATCH_OFF(k) (SCRATCH_SP_OFFSET + 4 * (k))
>> -
>> -#define SEEN_MEM ((1 << BPF_MEMWORDS) - 1)
>> -#define SEEN_MEM_WORD(k) (1 << (k))
>> -#define SEEN_X (1 << BPF_MEMWORDS)
>> -#define SEEN_CALL (1 << (BPF_MEMWORDS + 1))
>> -#define SEEN_SKB (1 << (BPF_MEMWORDS + 2))
>> -#define SEEN_DATA (1 << (BPF_MEMWORDS + 3))
>> +#define dst_lo dst[1]
>> +#define dst_hi dst[0]
>> +#define src_lo src[1]
>> +#define src_hi src[0]
>>
>> -#define FLAG_NEED_X_RESET (1 << 0)
>> -#define FLAG_IMM_OVERFLOW (1 << 1)
>> +/*
>> + * JIT Context:
>> + *
>> + * prog : bpf_prog
>> + * idx : index of current last JITed instruction.
>> + * prologue_bytes : bytes used in prologue.
>> + * epilogue_offset : offset of epilogue starting.
>> + * seen : bit mask used for JIT optimization.
>> + * offsets : array of eBPF instruction offsets in
>> + * JITed code.
>> + * target : final JITed code.
>> + * epilogue_bytes : no of bytes used in epilogue.
>> + * imm_count : no of immediate counts used for global
>> + * variables.
>> + * imms : array of global variable addresses.
>> + */
>>
>> struct jit_ctx {
>> - const struct bpf_prog *skf;
>> - unsigned idx;
>> - unsigned prologue_bytes;
>> - int ret0_fp_idx;
>> + const struct bpf_prog *prog;
>> + unsigned int idx;
>> + unsigned int prologue_bytes;
>> + unsigned int epilogue_offset;
>> u32 seen;
>> u32 flags;
>> u32 *offsets;
>> @@ -72,68 +121,16 @@ struct jit_ctx {
>> #endif
>> };
>>
>> -int bpf_jit_enable __read_mostly;
>> -
>> -static inline int call_neg_helper(struct sk_buff *skb, int offset, void *ret,
>> - unsigned int size)
>> -{
>> - void *ptr = bpf_internal_load_pointer_neg_helper(skb, offset, size);
>> -
>> - if (!ptr)
>> - return -EFAULT;
>> - memcpy(ret, ptr, size);
>> - return 0;
>> -}
>> -
>> -static u64 jit_get_skb_b(struct sk_buff *skb, int offset)
>> -{
>> - u8 ret;
>> - int err;
>> -
>> - if (offset < 0)
>> - err = call_neg_helper(skb, offset, &ret, 1);
>> - else
>> - err = skb_copy_bits(skb, offset, &ret, 1);
>> -
>> - return (u64)err << 32 | ret;
>> -}
>> -
>> -static u64 jit_get_skb_h(struct sk_buff *skb, int offset)
>> -{
>> - u16 ret;
>> - int err;
>> -
>> - if (offset < 0)
>> - err = call_neg_helper(skb, offset, &ret, 2);
>> - else
>> - err = skb_copy_bits(skb, offset, &ret, 2);
>> -
>> - return (u64)err << 32 | ntohs(ret);
>> -}
>> -
>> -static u64 jit_get_skb_w(struct sk_buff *skb, int offset)
>> -{
>> - u32 ret;
>> - int err;
>> -
>> - if (offset < 0)
>> - err = call_neg_helper(skb, offset, &ret, 4);
>> - else
>> - err = skb_copy_bits(skb, offset, &ret, 4);
>> -
>> - return (u64)err << 32 | ntohl(ret);
>> -}
>> -
>> /*
>> * Wrappers which handle both OABI and EABI and assures Thumb2 interworking
>> * (where the assembly routines like __aeabi_uidiv could cause problems).
>> */
>> -static u32 jit_udiv(u32 dividend, u32 divisor)
>> +static u32 jit_udiv32(u32 dividend, u32 divisor)
>> {
>> return dividend / divisor;
>> }
>>
>> -static u32 jit_mod(u32 dividend, u32 divisor)
>> +static u32 jit_mod32(u32 dividend, u32 divisor)
>> {
>> return dividend % divisor;
>> }
>> @@ -157,36 +154,22 @@ static inline void emit(u32 inst, struct jit_ctx *ctx)
>> _emit(ARM_COND_AL, inst, ctx);
>> }
>>
>> -static u16 saved_regs(struct jit_ctx *ctx)
>> +/*
>> + * Checks if immediate value can be converted to imm12(12 bits) value.
>> + */
>> +static int16_t imm8m(u32 x)
>> {
>> - u16 ret = 0;
>> -
>> - if ((ctx->skf->len > 1) ||
>> - (ctx->skf->insns[0].code == (BPF_RET | BPF_A)))
>> - ret |= 1 << r_A;
>> -
>> -#ifdef CONFIG_FRAME_POINTER
>> - ret |= (1 << ARM_FP) | (1 << ARM_IP) | (1 << ARM_LR) | (1 << ARM_PC);
>> -#else
>> - if (ctx->seen & SEEN_CALL)
>> - ret |= 1 << ARM_LR;
>> -#endif
>> - if (ctx->seen & (SEEN_DATA | SEEN_SKB))
>> - ret |= 1 << r_skb;
>> - if (ctx->seen & SEEN_DATA)
>> - ret |= (1 << r_skb_data) | (1 << r_skb_hl);
>> - if (ctx->seen & SEEN_X)
>> - ret |= 1 << r_X;
>> -
>> - return ret;
>> -}
>> + u32 rot;
>>
>> -static inline int mem_words_used(struct jit_ctx *ctx)
>> -{
>> - /* yes, we do waste some stack space IF there are "holes" in the set" */
>> - return fls(ctx->seen & SEEN_MEM);
>> + for (rot = 0; rot < 16; rot++)
>> + if ((x & ~ror32(0xff, 2 * rot)) == 0)
>> + return rol32(x, 2 * rot) | (rot << 8);
>> + return -1;
>> }
>>
>> +/*
>> + * Initializes the JIT space with undefined instructions.
>> + */
>> static void jit_fill_hole(void *area, unsigned int size)
>> {
>> u32 *ptr;
>> @@ -195,88 +178,34 @@ static void jit_fill_hole(void *area, unsigned int size)
>> *ptr++ = __opcode_to_mem_arm(ARM_INST_UDF);
>> }
>>
>> -static void build_prologue(struct jit_ctx *ctx)
>> -{
>> - u16 reg_set = saved_regs(ctx);
>> - u16 off;
>> -
>> -#ifdef CONFIG_FRAME_POINTER
>> - emit(ARM_MOV_R(ARM_IP, ARM_SP), ctx);
>> - emit(ARM_PUSH(reg_set), ctx);
>> - emit(ARM_SUB_I(ARM_FP, ARM_IP, 4), ctx);
>> -#else
>> - if (reg_set)
>> - emit(ARM_PUSH(reg_set), ctx);
>> -#endif
>> +/* Stack must be multiples of 16 Bytes */
>> +#define STACK_ALIGN(sz) (((sz) + 15) & ~15)
>>
>> - if (ctx->seen & (SEEN_DATA | SEEN_SKB))
>> - emit(ARM_MOV_R(r_skb, ARM_R0), ctx);
>> -
>> - if (ctx->seen & SEEN_DATA) {
>> - off = offsetof(struct sk_buff, data);
>> - emit(ARM_LDR_I(r_skb_data, r_skb, off), ctx);
>> - /* headlen = len - data_len */
>> - off = offsetof(struct sk_buff, len);
>> - emit(ARM_LDR_I(r_skb_hl, r_skb, off), ctx);
>> - off = offsetof(struct sk_buff, data_len);
>> - emit(ARM_LDR_I(r_scratch, r_skb, off), ctx);
>> - emit(ARM_SUB_R(r_skb_hl, r_skb_hl, r_scratch), ctx);
>> - }
>> -
>> - if (ctx->flags & FLAG_NEED_X_RESET)
>> - emit(ARM_MOV_I(r_X, 0), ctx);
>> -
>> - /* do not leak kernel data to userspace */
>> - if (bpf_needs_clear_a(&ctx->skf->insns[0]))
>> - emit(ARM_MOV_I(r_A, 0), ctx);
>> -
>> - /* stack space for the BPF_MEM words */
>> - if (ctx->seen & SEEN_MEM)
>> - emit(ARM_SUB_I(ARM_SP, ARM_SP, mem_words_used(ctx) * 4), ctx);
>> -}
>> -
>> -static void build_epilogue(struct jit_ctx *ctx)
>> -{
>> - u16 reg_set = saved_regs(ctx);
>> -
>> - if (ctx->seen & SEEN_MEM)
>> - emit(ARM_ADD_I(ARM_SP, ARM_SP, mem_words_used(ctx) * 4), ctx);
>> -
>> - reg_set &= ~(1 << ARM_LR);
>> -
>> -#ifdef CONFIG_FRAME_POINTER
>> - /* the first instruction of the prologue was: mov ip, sp */
>> - reg_set &= ~(1 << ARM_IP);
>> - reg_set |= (1 << ARM_SP);
>> - emit(ARM_LDM(ARM_SP, reg_set), ctx);
>> -#else
>> - if (reg_set) {
>> - if (ctx->seen & SEEN_CALL)
>> - reg_set |= 1 << ARM_PC;
>> - emit(ARM_POP(reg_set), ctx);
>> - }
>> +/* Stack space for BPF_REG_2, BPF_REG_3, BPF_REG_4,
>> + * BPF_REG_5, BPF_REG_7, BPF_REG_8, BPF_REG_9,
>> + * BPF_REG_FP and Tail call counts.
>> + */
>> +#define SCRATCH_SIZE 80
>>
>> - if (!(ctx->seen & SEEN_CALL))
>> - emit(ARM_BX(ARM_LR), ctx);
>> -#endif
>> -}
>> +/* total stack size used in JITed code */
>> +#define _STACK_SIZE \
>> + (MAX_BPF_STACK + \
>> + + SCRATCH_SIZE + \
>> + + 4 /* extra for skb_copy_bits buffer */)
>>
>> -static int16_t imm8m(u32 x)
>> -{
>> - u32 rot;
>> +#define STACK_SIZE STACK_ALIGN(_STACK_SIZE)
>>
>> - for (rot = 0; rot < 16; rot++)
>> - if ((x & ~ror32(0xff, 2 * rot)) == 0)
>> - return rol32(x, 2 * rot) | (rot << 8);
>> +/* Get the offset of eBPF REGISTERs stored on scratch space. */
>> +#define STACK_VAR(off) (STACK_SIZE-off-4)
>>
>> - return -1;
>> -}
>> +/* Offset of skb_copy_bits buffer */
>> +#define SKB_BUFFER STACK_VAR(SCRATCH_SIZE)
>>
>> #if __LINUX_ARM_ARCH__ < 7
>>
>> static u16 imm_offset(u32 k, struct jit_ctx *ctx)
>> {
>> - unsigned i = 0, offset;
>> + unsigned int i = 0, offset;
>> u16 imm;
>>
>> /* on the "fake" run we just count them (duplicates included) */
>> @@ -295,7 +224,7 @@ static u16 imm_offset(u32 k, struct jit_ctx *ctx)
>> ctx->imms[i] = k;
>>
>> /* constants go just after the epilogue */
>> - offset = ctx->offsets[ctx->skf->len];
>> + offset = ctx->offsets[ctx->prog->len - 1] * 4;
>> offset += ctx->prologue_bytes;
>> offset += ctx->epilogue_bytes;
>> offset += i * 4;
>> @@ -319,10 +248,22 @@ static u16 imm_offset(u32 k, struct jit_ctx *ctx)
>>
>> #endif /* __LINUX_ARM_ARCH__ */
>>
>> +static inline int bpf2a32_offset(int bpf_to, int bpf_from,
>> + const struct jit_ctx *ctx) {
>> + int to, from;
>> +
>> + if (ctx->target == NULL)
>> + return 0;
>> + to = ctx->offsets[bpf_to];
>> + from = ctx->offsets[bpf_from];
>> +
>> + return to - from - 1;
>> +}
>> +
>> /*
>> * Move an immediate that's not an imm8m to a core register.
>> */
>> -static inline void emit_mov_i_no8m(int rd, u32 val, struct jit_ctx *ctx)
>> +static inline void emit_mov_i_no8m(const u8 rd, u32 val, struct jit_ctx *ctx)
>> {
>> #if __LINUX_ARM_ARCH__ < 7
>> emit(ARM_LDR_I(rd, ARM_PC, imm_offset(val, ctx)), ctx);
>> @@ -333,7 +274,7 @@ static inline void emit_mov_i_no8m(int rd, u32 val, struct jit_ctx *ctx)
>> #endif
>> }
>>
>> -static inline void emit_mov_i(int rd, u32 val, struct jit_ctx *ctx)
>> +static inline void emit_mov_i(const u8 rd, u32 val, struct jit_ctx *ctx)
>> {
>> int imm12 = imm8m(val);
>>
>> @@ -343,676 +284,1553 @@ static inline void emit_mov_i(int rd, u32 val, struct jit_ctx *ctx)
>> emit_mov_i_no8m(rd, val, ctx);
>> }
>>
>> -#if __LINUX_ARM_ARCH__ < 6
>> -
>> -static void emit_load_be32(u8 cond, u8 r_res, u8 r_addr, struct jit_ctx *ctx)
>> +static inline void emit_blx_r(u8 tgt_reg, struct jit_ctx *ctx)
>> {
>> - _emit(cond, ARM_LDRB_I(ARM_R3, r_addr, 1), ctx);
>> - _emit(cond, ARM_LDRB_I(ARM_R1, r_addr, 0), ctx);
>> - _emit(cond, ARM_LDRB_I(ARM_R2, r_addr, 3), ctx);
>> - _emit(cond, ARM_LSL_I(ARM_R3, ARM_R3, 16), ctx);
>> - _emit(cond, ARM_LDRB_I(ARM_R0, r_addr, 2), ctx);
>> - _emit(cond, ARM_ORR_S(ARM_R3, ARM_R3, ARM_R1, SRTYPE_LSL, 24), ctx);
>> - _emit(cond, ARM_ORR_R(ARM_R3, ARM_R3, ARM_R2), ctx);
>> - _emit(cond, ARM_ORR_S(r_res, ARM_R3, ARM_R0, SRTYPE_LSL, 8), ctx);
>> + ctx->seen |= SEEN_CALL;
>> +#if __LINUX_ARM_ARCH__ < 5
>> + emit(ARM_MOV_R(ARM_LR, ARM_PC), ctx);
>> +
>> + if (elf_hwcap & HWCAP_THUMB)
>> + emit(ARM_BX(tgt_reg), ctx);
>> + else
>> + emit(ARM_MOV_R(ARM_PC, tgt_reg), ctx);
>> +#else
>> + emit(ARM_BLX_R(tgt_reg), ctx);
>> +#endif
>> }
>>
>> -static void emit_load_be16(u8 cond, u8 r_res, u8 r_addr, struct jit_ctx *ctx)
>> +static inline int epilogue_offset(const struct jit_ctx *ctx)
>> {
>> - _emit(cond, ARM_LDRB_I(ARM_R1, r_addr, 0), ctx);
>> - _emit(cond, ARM_LDRB_I(ARM_R2, r_addr, 1), ctx);
>> - _emit(cond, ARM_ORR_S(r_res, ARM_R2, ARM_R1, SRTYPE_LSL, 8), ctx);
>> + int to, from;
>> + /* No need for 1st dummy run */
>> + if (ctx->target == NULL)
>> + return 0;
>> + to = ctx->epilogue_offset;
>> + from = ctx->idx;
>> +
>> + return to - from - 2;
>> }
>>
>> -static inline void emit_swap16(u8 r_dst, u8 r_src, struct jit_ctx *ctx)
>> +static inline void emit_udivmod(u8 rd, u8 rm, u8 rn, struct jit_ctx *ctx, u8 op)
>> {
>> - /* r_dst = (r_src << 8) | (r_src >> 8) */
>> - emit(ARM_LSL_I(ARM_R1, r_src, 8), ctx);
>> - emit(ARM_ORR_S(r_dst, ARM_R1, r_src, SRTYPE_LSR, 8), ctx);
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + s32 jmp_offset;
>> +
>> + /* checks if divisor is zero or not. If it is, then
>> + * exit directly.
>> + */
>> + emit(ARM_CMP_I(rn, 0), ctx);
>> + _emit(ARM_COND_EQ, ARM_MOV_I(ARM_R0, 0), ctx);
>> + jmp_offset = epilogue_offset(ctx);
>> + _emit(ARM_COND_EQ, ARM_B(jmp_offset), ctx);
>> +#if __LINUX_ARM_ARCH__ == 7
>> + if (elf_hwcap & HWCAP_IDIVA) {
>> + if (op == BPF_DIV)
>> + emit(ARM_UDIV(rd, rm, rn), ctx);
>> + else {
>> + emit(ARM_UDIV(ARM_IP, rm, rn), ctx);
>> + emit(ARM_MLS(rd, rn, ARM_IP, rm), ctx);
>> + }
>> + return;
>> + }
>> +#endif
>>
>> /*
>> - * we need to mask out the bits set in r_dst[23:16] due to
>> - * the first shift instruction.
>> - *
>> - * note that 0x8ff is the encoded immediate 0x00ff0000.
>> + * For BPF_ALU | BPF_DIV | BPF_K instructions
>> + * As ARM_R1 and ARM_R0 contains 1st argument of bpf
>> + * function, we need to save it on caller side to save
>> + * it from getting destroyed within callee.
>> + * After the return from the callee, we restore ARM_R0
>> + * ARM_R1.
>> */
>> - emit(ARM_BIC_I(r_dst, r_dst, 0x8ff), ctx);
>> -}
>> + if (rn != ARM_R1) {
>> + emit(ARM_MOV_R(tmp[0], ARM_R1), ctx);
>> + emit(ARM_MOV_R(ARM_R1, rn), ctx);
>> + }
>> + if (rm != ARM_R0) {
>> + emit(ARM_MOV_R(tmp[1], ARM_R0), ctx);
>> + emit(ARM_MOV_R(ARM_R0, rm), ctx);
>> + }
>>
>> -#else /* ARMv6+ */
>> + /* Call appropriate function */
>> + ctx->seen |= SEEN_CALL;
>> + emit_mov_i(ARM_IP, op == BPF_DIV ?
>> + (u32)jit_udiv32 : (u32)jit_mod32, ctx);
>> + emit_blx_r(ARM_IP, ctx);
>>
>> -static void emit_load_be32(u8 cond, u8 r_res, u8 r_addr, struct jit_ctx *ctx)
>> -{
>> - _emit(cond, ARM_LDR_I(r_res, r_addr, 0), ctx);
>> -#ifdef __LITTLE_ENDIAN
>> - _emit(cond, ARM_REV(r_res, r_res), ctx);
>> -#endif
>> + /* Save return value */
>> + if (rd != ARM_R0)
>> + emit(ARM_MOV_R(rd, ARM_R0), ctx);
>> +
>> + /* Restore ARM_R0 and ARM_R1 */
>> + if (rn != ARM_R1)
>> + emit(ARM_MOV_R(ARM_R1, tmp[0]), ctx);
>> + if (rm != ARM_R0)
>> + emit(ARM_MOV_R(ARM_R0, tmp[1]), ctx);
>> }
>>
>> -static void emit_load_be16(u8 cond, u8 r_res, u8 r_addr, struct jit_ctx *ctx)
>> +/* Checks whether BPF register is on scratch stack space or not. */
>> +static inline bool is_on_stack(u8 bpf_reg)
>> {
>> - _emit(cond, ARM_LDRH_I(r_res, r_addr, 0), ctx);
>> -#ifdef __LITTLE_ENDIAN
>> - _emit(cond, ARM_REV16(r_res, r_res), ctx);
>> -#endif
>> + static u8 stack_regs[] = {BPF_REG_AX, BPF_REG_3, BPF_REG_4, BPF_REG_5,
>> + BPF_REG_7, BPF_REG_8, BPF_REG_9, TCALL_CNT,
>> + BPF_REG_2, BPF_REG_FP};
>> + int i, reg_len = sizeof(stack_regs);
>> +
>> + for (i = 0 ; i < reg_len ; i++) {
>> + if (bpf_reg == stack_regs[i])
>> + return true;
>> + }
>> + return false;
>> }
>>
>> -static inline void emit_swap16(u8 r_dst __maybe_unused,
>> - u8 r_src __maybe_unused,
>> - struct jit_ctx *ctx __maybe_unused)
>> +static inline void emit_a32_mov_i(const u8 dst, const u32 val,
>> + bool dstk, struct jit_ctx *ctx)
>> {
>> -#ifdef __LITTLE_ENDIAN
>> - emit(ARM_REV16(r_dst, r_src), ctx);
>> -#endif
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> +
>> + if (dstk) {
>> + emit_mov_i(tmp[1], val, ctx);
>> + emit(ARM_STR_I(tmp[1], ARM_SP, STACK_VAR(dst)), ctx);
>> + } else {
>> + emit_mov_i(dst, val, ctx);
>> + }
>> }
>>
>> -#endif /* __LINUX_ARM_ARCH__ < 6 */
>> +/* Sign extended move */
>> +static inline void emit_a32_mov_i64(const bool is64, const u8 dst[],
>> + const u32 val, bool dstk,
>> + struct jit_ctx *ctx) {
>> + u32 hi = 0;
>>
>> + if (is64 && (val & (1<<31)))
>> + hi = (u32)~0;
>> + emit_a32_mov_i(dst_lo, val, dstk, ctx);
>> + emit_a32_mov_i(dst_hi, hi, dstk, ctx);
>> +}
>>
>> -/* Compute the immediate value for a PC-relative branch. */
>> -static inline u32 b_imm(unsigned tgt, struct jit_ctx *ctx)
>> -{
>> - u32 imm;
>> +static inline void emit_a32_add_r(const u8 dst, const u8 src,
>> + const bool is64, const bool hi,
>> + struct jit_ctx *ctx) {
>> + /* 64 bit :
>> + * adds dst_lo, dst_lo, src_lo
>> + * adc dst_hi, dst_hi, src_hi
>> + * 32 bit :
>> + * add dst_lo, dst_lo, src_lo
>> + */
>> + if (!hi && is64)
>> + emit(ARM_ADDS_R(dst, dst, src), ctx);
>> + else if (hi && is64)
>> + emit(ARM_ADC_R(dst, dst, src), ctx);
>> + else
>> + emit(ARM_ADD_R(dst, dst, src), ctx);
>> +}
>>
>> - if (ctx->target == NULL)
>> - return 0;
>> - /*
>> - * BPF allows only forward jumps and the offset of the target is
>> - * still the one computed during the first pass.
>> +static inline void emit_a32_sub_r(const u8 dst, const u8 src,
>> + const bool is64, const bool hi,
>> + struct jit_ctx *ctx) {
>> + /* 64 bit :
>> + * subs dst_lo, dst_lo, src_lo
>> + * sbc dst_hi, dst_hi, src_hi
>> + * 32 bit :
>> + * sub dst_lo, dst_lo, src_lo
>> */
>> - imm = ctx->offsets[tgt] + ctx->prologue_bytes - (ctx->idx * 4 + 8);
>> + if (!hi && is64)
>> + emit(ARM_SUBS_R(dst, dst, src), ctx);
>> + else if (hi && is64)
>> + emit(ARM_SBC_R(dst, dst, src), ctx);
>> + else
>> + emit(ARM_SUB_R(dst, dst, src), ctx);
>> +}
>>
>> - return imm >> 2;
>> +static inline void emit_alu_r(const u8 dst, const u8 src, const bool is64,
>> + const bool hi, const u8 op, struct jit_ctx *ctx){
>> + switch (BPF_OP(op)) {
>> + /* dst = dst + src */
>> + case BPF_ADD:
>> + emit_a32_add_r(dst, src, is64, hi, ctx);
>> + break;
>> + /* dst = dst - src */
>> + case BPF_SUB:
>> + emit_a32_sub_r(dst, src, is64, hi, ctx);
>> + break;
>> + /* dst = dst | src */
>> + case BPF_OR:
>> + emit(ARM_ORR_R(dst, dst, src), ctx);
>> + break;
>> + /* dst = dst & src */
>> + case BPF_AND:
>> + emit(ARM_AND_R(dst, dst, src), ctx);
>> + break;
>> + /* dst = dst ^ src */
>> + case BPF_XOR:
>> + emit(ARM_EOR_R(dst, dst, src), ctx);
>> + break;
>> + /* dst = dst * src */
>> + case BPF_MUL:
>> + emit(ARM_MUL(dst, dst, src), ctx);
>> + break;
>> + /* dst = dst << src */
>> + case BPF_LSH:
>> + emit(ARM_LSL_R(dst, dst, src), ctx);
>> + break;
>> + /* dst = dst >> src */
>> + case BPF_RSH:
>> + emit(ARM_LSR_R(dst, dst, src), ctx);
>> + break;
>> + /* dst = dst >> src (signed)*/
>> + case BPF_ARSH:
>> + emit(ARM_MOV_SR(dst, dst, SRTYPE_ASR, src), ctx);
>> + break;
>> + }
>> }
>>
>> -#define OP_IMM3(op, r1, r2, imm_val, ctx) \
>> - do { \
>> - imm12 = imm8m(imm_val); \
>> - if (imm12 < 0) { \
>> - emit_mov_i_no8m(r_scratch, imm_val, ctx); \
>> - emit(op ## _R((r1), (r2), r_scratch), ctx); \
>> - } else { \
>> - emit(op ## _I((r1), (r2), imm12), ctx); \
>> - } \
>> - } while (0)
>> -
>> -static inline void emit_err_ret(u8 cond, struct jit_ctx *ctx)
>> -{
>> - if (ctx->ret0_fp_idx >= 0) {
>> - _emit(cond, ARM_B(b_imm(ctx->ret0_fp_idx, ctx)), ctx);
>> - /* NOP to keep the size constant between passes */
>> - emit(ARM_MOV_R(ARM_R0, ARM_R0), ctx);
>> +/* ALU operation (32 bit)
>> + * dst = dst (op) src
>> + */
>> +static inline void emit_a32_alu_r(const u8 dst, const u8 src,
>> + bool dstk, bool sstk,
>> + struct jit_ctx *ctx, const bool is64,
>> + const bool hi, const u8 op) {
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + u8 rn = sstk ? tmp[1] : src;
>> +
>> + if (sstk)
>> + emit(ARM_LDR_I(rn, ARM_SP, STACK_VAR(src)), ctx);
>> +
>> + /* ALU operation */
>> + if (dstk) {
>> + emit(ARM_LDR_I(tmp[0], ARM_SP, STACK_VAR(dst)), ctx);
>> + emit_alu_r(tmp[0], rn, is64, hi, op, ctx);
>> + emit(ARM_STR_I(tmp[0], ARM_SP, STACK_VAR(dst)), ctx);
>> } else {
>> - _emit(cond, ARM_MOV_I(ARM_R0, 0), ctx);
>> - _emit(cond, ARM_B(b_imm(ctx->skf->len, ctx)), ctx);
>> + emit_alu_r(dst, rn, is64, hi, op, ctx);
>> }
>> }
>>
>> -static inline void emit_blx_r(u8 tgt_reg, struct jit_ctx *ctx)
>> -{
>> -#if __LINUX_ARM_ARCH__ < 5
>> - emit(ARM_MOV_R(ARM_LR, ARM_PC), ctx);
>> +/* ALU operation (64 bit) */
>> +static inline void emit_a32_alu_r64(const bool is64, const u8 dst[],
>> + const u8 src[], bool dstk,
>> + bool sstk, struct jit_ctx *ctx,
>> + const u8 op) {
>> + emit_a32_alu_r(dst_lo, src_lo, dstk, sstk, ctx, is64, false, op);
>> + if (is64)
>> + emit_a32_alu_r(dst_hi, src_hi, dstk, sstk, ctx, is64, true, op);
>> + else
>> + emit_a32_mov_i(dst_hi, 0, dstk, ctx);
>> +}
>>
>> - if (elf_hwcap & HWCAP_THUMB)
>> - emit(ARM_BX(tgt_reg), ctx);
>> +/* dst = imm (4 bytes)*/
>> +static inline void emit_a32_mov_r(const u8 dst, const u8 src,
>> + bool dstk, bool sstk,
>> + struct jit_ctx *ctx) {
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + u8 rt = sstk ? tmp[0] : src;
>> +
>> + if (sstk)
>> + emit(ARM_LDR_I(tmp[0], ARM_SP, STACK_VAR(src)), ctx);
>> + if (dstk)
>> + emit(ARM_STR_I(rt, ARM_SP, STACK_VAR(dst)), ctx);
>> else
>> - emit(ARM_MOV_R(ARM_PC, tgt_reg), ctx);
>> -#else
>> - emit(ARM_BLX_R(tgt_reg), ctx);
>> -#endif
>> + emit(ARM_MOV_R(dst, rt), ctx);
>> }
>>
>> -static inline void emit_udivmod(u8 rd, u8 rm, u8 rn, struct jit_ctx *ctx,
>> - int bpf_op)
>> -{
>> -#if __LINUX_ARM_ARCH__ == 7
>> - if (elf_hwcap & HWCAP_IDIVA) {
>> - if (bpf_op == BPF_DIV)
>> - emit(ARM_UDIV(rd, rm, rn), ctx);
>> - else {
>> - emit(ARM_UDIV(ARM_R3, rm, rn), ctx);
>> - emit(ARM_MLS(rd, rn, ARM_R3, rm), ctx);
>> - }
>> - return;
>> +/* dst = src */
>> +static inline void emit_a32_mov_r64(const bool is64, const u8 dst[],
>> + const u8 src[], bool dstk,
>> + bool sstk, struct jit_ctx *ctx) {
>> + emit_a32_mov_r(dst_lo, src_lo, dstk, sstk, ctx);
>> + if (is64) {
>> + /* complete 8 byte move */
>> + emit_a32_mov_r(dst_hi, src_hi, dstk, sstk, ctx);
>> + } else {
>> + /* Zero out high 4 bytes */
>> + emit_a32_mov_i(dst_hi, 0, dstk, ctx);
>> }
>> -#endif
>> +}
>>
>> - /*
>> - * For BPF_ALU | BPF_DIV | BPF_K instructions, rm is ARM_R4
>> - * (r_A) and rn is ARM_R0 (r_scratch) so load rn first into
>> - * ARM_R1 to avoid accidentally overwriting ARM_R0 with rm
>> - * before using it as a source for ARM_R1.
>> - *
>> - * For BPF_ALU | BPF_DIV | BPF_X rm is ARM_R4 (r_A) and rn is
>> - * ARM_R5 (r_X) so there is no particular register overlap
>> - * issues.
>> - */
>> - if (rn != ARM_R1)
>> - emit(ARM_MOV_R(ARM_R1, rn), ctx);
>> - if (rm != ARM_R0)
>> - emit(ARM_MOV_R(ARM_R0, rm), ctx);
>> +/* Shift operations */
>> +static inline void emit_a32_alu_i(const u8 dst, const u32 val, bool dstk,
>> + struct jit_ctx *ctx, const u8 op) {
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + u8 rd = dstk ? tmp[0] : dst;
>> +
>> + if (dstk)
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst)), ctx);
>> +
>> + /* Do shift operation */
>> + switch (op) {
>> + case BPF_LSH:
>> + emit(ARM_LSL_I(rd, rd, val), ctx);
>> + break;
>> + case BPF_RSH:
>> + emit(ARM_LSR_I(rd, rd, val), ctx);
>> + break;
>> + case BPF_NEG:
>> + emit(ARM_RSB_I(rd, rd, val), ctx);
>> + break;
>> + }
>> +
>> + if (dstk)
>> + emit(ARM_STR_I(rd, ARM_SP, STACK_VAR(dst)), ctx);
>> +}
>> +
>> +/* dst = ~dst (64 bit) */
>> +static inline void emit_a32_neg64(const u8 dst[], bool dstk,
>> + struct jit_ctx *ctx){
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + u8 rd = dstk ? tmp[1] : dst[1];
>> + u8 rm = dstk ? tmp[0] : dst[0];
>> +
>> + /* Setup Operand */
>> + if (dstk) {
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +
>> + /* Do Negate Operation */
>> + emit(ARM_RSBS_I(rd, rd, 0), ctx);
>> + emit(ARM_RSC_I(rm, rm, 0), ctx);
>> +
>> + if (dstk) {
>> + emit(ARM_STR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_STR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +}
>>
>> +/* dst = dst << src */
>> +static inline void emit_a32_lsh_r64(const u8 dst[], const u8 src[], bool dstk,
>> + bool sstk, struct jit_ctx *ctx) {
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> +
>> + /* Setup Operands */
>> + u8 rt = sstk ? tmp2[1] : src_lo;
>> + u8 rd = dstk ? tmp[1] : dst_lo;
>> + u8 rm = dstk ? tmp[0] : dst_hi;
>> +
>> + if (sstk)
>> + emit(ARM_LDR_I(rt, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + if (dstk) {
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +
>> + /* Do LSH operation */
>> + emit(ARM_SUB_I(ARM_IP, rt, 32), ctx);
>> + emit(ARM_RSB_I(tmp2[0], rt, 32), ctx);
>> + /* As we are using ARM_LR */
>> ctx->seen |= SEEN_CALL;
>> - emit_mov_i(ARM_R3, bpf_op == BPF_DIV ? (u32)jit_udiv : (u32)jit_mod,
>> - ctx);
>> - emit_blx_r(ARM_R3, ctx);
>> + emit(ARM_MOV_SR(ARM_LR, rm, SRTYPE_ASL, rt), ctx);
>> + emit(ARM_ORR_SR(ARM_LR, ARM_LR, rd, SRTYPE_ASL, ARM_IP), ctx);
>> + emit(ARM_ORR_SR(ARM_IP, ARM_LR, rd, SRTYPE_LSR, tmp2[0]), ctx);
>> + emit(ARM_MOV_SR(ARM_LR, rd, SRTYPE_ASL, rt), ctx);
>> +
>> + if (dstk) {
>> + emit(ARM_STR_I(ARM_LR, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_STR_I(ARM_IP, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + } else {
>> + emit(ARM_MOV_R(rd, ARM_LR), ctx);
>> + emit(ARM_MOV_R(rm, ARM_IP), ctx);
>> + }
>> +}
>>
>> - if (rd != ARM_R0)
>> - emit(ARM_MOV_R(rd, ARM_R0), ctx);
>> +/* dst = dst >> src (signed)*/
>> +static inline void emit_a32_arsh_r64(const u8 dst[], const u8 src[], bool dstk,
>> + bool sstk, struct jit_ctx *ctx) {
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> + /* Setup Operands */
>> + u8 rt = sstk ? tmp2[1] : src_lo;
>> + u8 rd = dstk ? tmp[1] : dst_lo;
>> + u8 rm = dstk ? tmp[0] : dst_hi;
>> +
>> + if (sstk)
>> + emit(ARM_LDR_I(rt, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + if (dstk) {
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +
>> + /* Do the ARSH operation */
>> + emit(ARM_RSB_I(ARM_IP, rt, 32), ctx);
>> + emit(ARM_SUBS_I(tmp2[0], rt, 32), ctx);
>> + /* As we are using ARM_LR */
>> + ctx->seen |= SEEN_CALL;
>> + emit(ARM_MOV_SR(ARM_LR, rd, SRTYPE_LSR, rt), ctx);
>> + emit(ARM_ORR_SR(ARM_LR, ARM_LR, rm, SRTYPE_ASL, ARM_IP), ctx);
>> + _emit(ARM_COND_MI, ARM_B(0), ctx);
>> + emit(ARM_ORR_SR(ARM_LR, ARM_LR, rm, SRTYPE_ASR, tmp2[0]), ctx);
>> + emit(ARM_MOV_SR(ARM_IP, rm, SRTYPE_ASR, rt), ctx);
>> + if (dstk) {
>> + emit(ARM_STR_I(ARM_LR, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_STR_I(ARM_IP, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + } else {
>> + emit(ARM_MOV_R(rd, ARM_LR), ctx);
>> + emit(ARM_MOV_R(rm, ARM_IP), ctx);
>> + }
>> }
>>
>> -static inline void update_on_xread(struct jit_ctx *ctx)
>> +/* dst = dst >> src */
>> +static inline void emit_a32_lsr_r64(const u8 dst[], const u8 src[], bool dstk,
>> + bool sstk, struct jit_ctx *ctx) {
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> + /* Setup Operands */
>> + u8 rt = sstk ? tmp2[1] : src_lo;
>> + u8 rd = dstk ? tmp[1] : dst_lo;
>> + u8 rm = dstk ? tmp[0] : dst_hi;
>> +
>> + if (sstk)
>> + emit(ARM_LDR_I(rt, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + if (dstk) {
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +
>> + /* Do LSH operation */
>> + emit(ARM_RSB_I(ARM_IP, rt, 32), ctx);
>> + emit(ARM_SUBS_I(tmp2[0], rt, 32), ctx);
>> + /* As we are using ARM_LR */
>> + ctx->seen |= SEEN_CALL;
>> + emit(ARM_MOV_SR(ARM_LR, rd, SRTYPE_LSR, rt), ctx);
>> + emit(ARM_ORR_SR(ARM_LR, ARM_LR, rm, SRTYPE_ASL, ARM_IP), ctx);
>> + emit(ARM_ORR_SR(ARM_LR, ARM_LR, rm, SRTYPE_LSR, tmp2[0]), ctx);
>> + emit(ARM_MOV_SR(ARM_IP, rm, SRTYPE_LSR, rt), ctx);
>> + if (dstk) {
>> + emit(ARM_STR_I(ARM_LR, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_STR_I(ARM_IP, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + } else {
>> + emit(ARM_MOV_R(rd, ARM_LR), ctx);
>> + emit(ARM_MOV_R(rm, ARM_IP), ctx);
>> + }
>> +}
>> +
>> +/* dst = dst << val */
>> +static inline void emit_a32_lsh_i64(const u8 dst[], bool dstk,
>> + const u32 val, struct jit_ctx *ctx){
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> + /* Setup operands */
>> + u8 rd = dstk ? tmp[1] : dst_lo;
>> + u8 rm = dstk ? tmp[0] : dst_hi;
>> +
>> + if (dstk) {
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +
>> + /* Do LSH operation */
>> + if (val < 32) {
>> + emit(ARM_MOV_SI(tmp2[0], rm, SRTYPE_ASL, val), ctx);
>> + emit(ARM_ORR_SI(rm, tmp2[0], rd, SRTYPE_LSR, 32 - val), ctx);
>> + emit(ARM_MOV_SI(rd, rd, SRTYPE_ASL, val), ctx);
>> + } else {
>> + if (val == 32)
>> + emit(ARM_MOV_R(rm, rd), ctx);
>> + else
>> + emit(ARM_MOV_SI(rm, rd, SRTYPE_ASL, val - 32), ctx);
>> + emit(ARM_EOR_R(rd, rd, rd), ctx);
>> + }
>> +
>> + if (dstk) {
>> + emit(ARM_STR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_STR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +}
>> +
>> +/* dst = dst >> val */
>> +static inline void emit_a32_lsr_i64(const u8 dst[], bool dstk,
>> + const u32 val, struct jit_ctx *ctx) {
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> + /* Setup operands */
>> + u8 rd = dstk ? tmp[1] : dst_lo;
>> + u8 rm = dstk ? tmp[0] : dst_hi;
>> +
>> + if (dstk) {
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +
>> + /* Do LSR operation */
>> + if (val < 32) {
>> + emit(ARM_MOV_SI(tmp2[1], rd, SRTYPE_LSR, val), ctx);
>> + emit(ARM_ORR_SI(rd, tmp2[1], rm, SRTYPE_ASL, 32 - val), ctx);
>> + emit(ARM_MOV_SI(rm, rm, SRTYPE_LSR, val), ctx);
>> + } else if (val == 32) {
>> + emit(ARM_MOV_R(rd, rm), ctx);
>> + emit(ARM_MOV_I(rm, 0), ctx);
>> + } else {
>> + emit(ARM_MOV_SI(rd, rm, SRTYPE_LSR, val - 32), ctx);
>> + emit(ARM_MOV_I(rm, 0), ctx);
>> + }
>> +
>> + if (dstk) {
>> + emit(ARM_STR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_STR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +}
>> +
>> +/* dst = dst >> val (signed) */
>> +static inline void emit_a32_arsh_i64(const u8 dst[], bool dstk,
>> + const u32 val, struct jit_ctx *ctx){
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> + /* Setup operands */
>> + u8 rd = dstk ? tmp[1] : dst_lo;
>> + u8 rm = dstk ? tmp[0] : dst_hi;
>> +
>> + if (dstk) {
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +
>> + /* Do ARSH operation */
>> + if (val < 32) {
>> + emit(ARM_MOV_SI(tmp2[1], rd, SRTYPE_LSR, val), ctx);
>> + emit(ARM_ORR_SI(rd, tmp2[1], rm, SRTYPE_ASL, 32 - val), ctx);
>> + emit(ARM_MOV_SI(rm, rm, SRTYPE_ASR, val), ctx);
>> + } else if (val == 32) {
>> + emit(ARM_MOV_R(rd, rm), ctx);
>> + emit(ARM_MOV_SI(rm, rm, SRTYPE_ASR, 31), ctx);
>> + } else {
>> + emit(ARM_MOV_SI(rd, rm, SRTYPE_ASR, val - 32), ctx);
>> + emit(ARM_MOV_SI(rm, rm, SRTYPE_ASR, 31), ctx);
>> + }
>> +
>> + if (dstk) {
>> + emit(ARM_STR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_STR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +}
>> +
>> +static inline void emit_a32_mul_r64(const u8 dst[], const u8 src[], bool dstk,
>> + bool sstk, struct jit_ctx *ctx) {
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> + /* Setup operands for multiplication */
>> + u8 rd = dstk ? tmp[1] : dst_lo;
>> + u8 rm = dstk ? tmp[0] : dst_hi;
>> + u8 rt = sstk ? tmp2[1] : src_lo;
>> + u8 rn = sstk ? tmp2[0] : src_hi;
>> +
>> + if (dstk) {
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> + if (sstk) {
>> + emit(ARM_LDR_I(rt, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + emit(ARM_LDR_I(rn, ARM_SP, STACK_VAR(src_hi)), ctx);
>> + }
>> +
>> + /* Do Multiplication */
>> + emit(ARM_MUL(ARM_IP, rd, rn), ctx);
>> + emit(ARM_MUL(ARM_LR, rm, rt), ctx);
>> + /* As we are using ARM_LR */
>> + ctx->seen |= SEEN_CALL;
>> + emit(ARM_ADD_R(ARM_LR, ARM_IP, ARM_LR), ctx);
>> +
>> + emit(ARM_UMULL(ARM_IP, rm, rd, rt), ctx);
>> + emit(ARM_ADD_R(rm, ARM_LR, rm), ctx);
>> + if (dstk) {
>> + emit(ARM_STR_I(ARM_IP, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_STR_I(rm, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + } else {
>> + emit(ARM_MOV_R(rd, ARM_IP), ctx);
>> + }
>> +}
>> +
>> +/* *(size *)(dst + off) = src */
>> +static inline void emit_str_r(const u8 dst, const u8 src, bool dstk,
>> + const s32 off, struct jit_ctx *ctx, const u8 sz){
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + u8 rd = dstk ? tmp[1] : dst;
>> +
>> + if (dstk)
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst)), ctx);
>> + if (off) {
>> + emit_a32_mov_i(tmp[0], off, false, ctx);
>> + emit(ARM_ADD_R(tmp[0], rd, tmp[0]), ctx);
>> + rd = tmp[0];
>> + }
>> + switch (sz) {
>> + case BPF_W:
>> + /* Store a Word */
>> + emit(ARM_STR_I(src, rd, 0), ctx);
>> + break;
>> + case BPF_H:
>> + /* Store a HalfWord */
>> + emit(ARM_STRH_I(src, rd, 0), ctx);
>> + break;
>> + case BPF_B:
>> + /* Store a Byte */
>> + emit(ARM_STRB_I(src, rd, 0), ctx);
>> + break;
>> + }
>> +}
>> +
>> +/* dst = *(size*)(src + off) */
>> +static inline void emit_ldx_r(const u8 dst, const u8 src, bool dstk,
>> + const s32 off, struct jit_ctx *ctx, const u8 sz){
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + u8 rd = dstk ? tmp[1] : dst;
>> + u8 rm = src;
>> +
>> + if (off) {
>> + emit_a32_mov_i(tmp[0], off, false, ctx);
>> + emit(ARM_ADD_R(tmp[0], tmp[0], src), ctx);
>> + rm = tmp[0];
>> + }
>> + switch (sz) {
>> + case BPF_W:
>> + /* Load a Word */
>> + emit(ARM_LDR_I(rd, rm, 0), ctx);
>> + break;
>> + case BPF_H:
>> + /* Load a HalfWord */
>> + emit(ARM_LDRH_I(rd, rm, 0), ctx);
>> + break;
>> + case BPF_B:
>> + /* Load a Byte */
>> + emit(ARM_LDRB_I(rd, rm, 0), ctx);
>> + break;
>> + }
>> + if (dstk)
>> + emit(ARM_STR_I(rd, ARM_SP, STACK_VAR(dst)), ctx);
>> +}
>> +
>> +/* Arithmatic Operation */
>> +static inline void emit_ar_r(const u8 rd, const u8 rt, const u8 rm,
>> + const u8 rn, struct jit_ctx *ctx, u8 op) {
>> + switch (op) {
>> + case BPF_JSET:
>> + ctx->seen |= SEEN_CALL;
>> + emit(ARM_AND_R(ARM_IP, rt, rn), ctx);
>> + emit(ARM_AND_R(ARM_LR, rd, rm), ctx);
>> + emit(ARM_ORRS_R(ARM_IP, ARM_LR, ARM_IP), ctx);
>> + break;
>> + case BPF_JEQ:
>> + case BPF_JNE:
>> + case BPF_JGT:
>> + case BPF_JGE:
>> + emit(ARM_CMP_R(rd, rm), ctx);
>> + _emit(ARM_COND_EQ, ARM_CMP_R(rt, rn), ctx);
>> + break;
>> + case BPF_JSGT:
>> + emit(ARM_CMP_R(rn, rt), ctx);
>> + emit(ARM_SBCS_R(ARM_IP, rm, rd), ctx);
>> + break;
>> + case BPF_JSGE:
>> + emit(ARM_CMP_R(rt, rn), ctx);
>> + emit(ARM_SBCS_R(ARM_IP, rd, rm), ctx);
>> + break;
>> + }
>> +}
>> +
>> +static int out_offset = -1; /* initialized on the first pass of build_body() */
>> +static int emit_bpf_tail_call(struct jit_ctx *ctx)
>> +{
>> +
>> + /* bpf_tail_call(void *prog_ctx, struct bpf_array *array, u64 index) */
>> + const u8 *r2 = bpf2a32[BPF_REG_2];
>> + const u8 *r3 = bpf2a32[BPF_REG_3];
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> + const u8 *tcc = bpf2a32[TCALL_CNT];
>> + const int idx0 = ctx->idx;
>> +#define cur_offset (ctx->idx - idx0)
>> +#define jmp_offset (out_offset - (cur_offset))
>> + u32 off, lo, hi;
>> +
>> + /* if (index >= array->map.max_entries)
>> + * goto out;
>> + */
>> + off = offsetof(struct bpf_array, map.max_entries);
>> + /* array->map.max_entries */
>> + emit_a32_mov_i(tmp[1], off, false, ctx);
>> + emit(ARM_LDR_I(tmp2[1], ARM_SP, STACK_VAR(r2[1])), ctx);
>> + emit(ARM_LDR_R(tmp[1], tmp2[1], tmp[1]), ctx);
>> + /* index (64 bit) */
>> + emit(ARM_LDR_I(tmp2[1], ARM_SP, STACK_VAR(r3[1])), ctx);
>> + /* index >= array->map.max_entries */
>> + emit(ARM_CMP_R(tmp2[1], tmp[1]), ctx);
>> + _emit(ARM_COND_CS, ARM_B(jmp_offset), ctx);
>> +
>> + /* if (tail_call_cnt > MAX_TAIL_CALL_CNT)
>> + * goto out;
>> + * tail_call_cnt++;
>> + */
>> + lo = (u32)MAX_TAIL_CALL_CNT;
>> + hi = (u32)((u64)MAX_TAIL_CALL_CNT >> 32);
>> + emit(ARM_LDR_I(tmp[1], ARM_SP, STACK_VAR(tcc[1])), ctx);
>> + emit(ARM_LDR_I(tmp[0], ARM_SP, STACK_VAR(tcc[0])), ctx);
>> + emit(ARM_CMP_I(tmp[0], hi), ctx);
>> + _emit(ARM_COND_EQ, ARM_CMP_I(tmp[1], lo), ctx);
>> + _emit(ARM_COND_HI, ARM_B(jmp_offset), ctx);
>> + emit(ARM_ADDS_I(tmp[1], tmp[1], 1), ctx);
>> + emit(ARM_ADC_I(tmp[0], tmp[0], 0), ctx);
>> + emit(ARM_STR_I(tmp[1], ARM_SP, STACK_VAR(tcc[1])), ctx);
>> + emit(ARM_STR_I(tmp[0], ARM_SP, STACK_VAR(tcc[0])), ctx);
>> +
>> + /* prog = array->ptrs[index]
>> + * if (prog == NULL)
>> + * goto out;
>> + */
>> + off = offsetof(struct bpf_array, ptrs);
>> + emit_a32_mov_i(tmp[1], off, false, ctx);
>> + emit(ARM_LDR_I(tmp2[1], ARM_SP, STACK_VAR(r2[1])), ctx);
>> + emit(ARM_LDR_R(tmp[1], tmp2[1], tmp[1]), ctx);
>> + emit(ARM_LDR_I(tmp2[1], ARM_SP, STACK_VAR(r3[1])), ctx);
>> + emit(ARM_MOV_SI(tmp[0], tmp2[1], SRTYPE_ASL, 2), ctx);
>> + emit(ARM_LDR_R(tmp[1], tmp[1], tmp[0]), ctx);
>> + emit(ARM_CMP_I(tmp[1], 0), ctx);
>> + _emit(ARM_COND_EQ, ARM_B(jmp_offset), ctx);
>> +
>> + /* goto *(prog->bpf_func + prologue_size); */
>> + off = offsetof(struct bpf_prog, bpf_func);
>> + emit_a32_mov_i(tmp2[1], off, false, ctx);
>> + emit(ARM_LDR_R(tmp[1], tmp[1], tmp2[1]), ctx);
>> + emit(ARM_ADD_I(tmp[1], tmp[1], ctx->prologue_bytes), ctx);
>> + emit(ARM_BX(tmp[1]), ctx);
>> +
>> + /* out: */
>> + if (out_offset == -1)
>> + out_offset = cur_offset;
>> + if (cur_offset != out_offset) {
>> + pr_err_once("tail_call out_offset = %d, expected %d!\n",
>> + cur_offset, out_offset);
>> + return -1;
>> + }
>> + return 0;
>> +#undef cur_offset
>> +#undef jmp_offset
>> +}
>> +
>> +/* 0xabcd => 0xcdab */
>> +static inline void emit_rev16(const u8 rd, const u8 rn, struct jit_ctx *ctx)
>> {
>> - if (!(ctx->seen & SEEN_X))
>> - ctx->flags |= FLAG_NEED_X_RESET;
>> +#if __LINUX_ARM_ARCH__ < 6
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> +
>> + emit(ARM_AND_I(tmp2[1], rn, 0xff), ctx);
>> + emit(ARM_MOV_SI(tmp2[0], rn, SRTYPE_LSR, 8), ctx);
>> + emit(ARM_AND_I(tmp2[0], tmp2[0], 0xff), ctx);
>> + emit(ARM_ORR_SI(rd, tmp2[0], tmp2[1], SRTYPE_LSL, 8), ctx);
>> +#else /* ARMv6+ */
>> + emit(ARM_REV16(rd, rn), ctx);
>> +#endif
>> +}
>>
>> - ctx->seen |= SEEN_X;
>> +/* 0xabcdefgh => 0xghefcdab */
>> +static inline void emit_rev32(const u8 rd, const u8 rn, struct jit_ctx *ctx)
>> +{
>> +#if __LINUX_ARM_ARCH__ < 6
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> +
>> + emit(ARM_AND_I(tmp2[1], rn, 0xff), ctx);
>> + emit(ARM_MOV_SI(tmp2[0], rn, SRTYPE_LSR, 24), ctx);
>> + emit(ARM_ORR_SI(ARM_IP, tmp2[0], tmp2[1], SRTYPE_LSL, 24), ctx);
>> +
>> + emit(ARM_MOV_SI(tmp2[1], rn, SRTYPE_LSR, 8), ctx);
>> + emit(ARM_AND_I(tmp2[1], tmp2[1], 0xff), ctx);
>> + emit(ARM_MOV_SI(tmp2[0], rn, SRTYPE_LSR, 16), ctx);
>> + emit(ARM_AND_I(tmp2[0], tmp2[0], 0xff), ctx);
>> + emit(ARM_MOV_SI(tmp2[0], tmp2[0], SRTYPE_LSL, 8), ctx);
>> + emit(ARM_ORR_SI(tmp2[0], tmp2[0], tmp2[1], SRTYPE_LSL, 16), ctx);
>> + emit(ARM_ORR_R(rd, ARM_IP, tmp2[0]), ctx);
>> +
>> +#else /* ARMv6+ */
>> + emit(ARM_REV(rd, rn), ctx);
>> +#endif
>> }
>>
>> -static int build_body(struct jit_ctx *ctx)
>> +static void build_prologue(struct jit_ctx *ctx)
>> {
>> - void *load_func[] = {jit_get_skb_b, jit_get_skb_h, jit_get_skb_w};
>> - const struct bpf_prog *prog = ctx->skf;
>> - const struct sock_filter *inst;
>> - unsigned i, load_order, off, condt;
>> - int imm12;
>> - u32 k;
>> + const u8 r0 = bpf2a32[BPF_REG_0][1];
>> + const u8 r2 = bpf2a32[BPF_REG_1][1];
>> + const u8 r3 = bpf2a32[BPF_REG_1][0];
>> + const u8 r4 = bpf2a32[BPF_REG_6][1];
>> + const u8 r5 = bpf2a32[BPF_REG_6][0];
>> + const u8 r6 = bpf2a32[TMP_REG_1][1];
>> + const u8 r7 = bpf2a32[TMP_REG_1][0];
>> + const u8 r8 = bpf2a32[TMP_REG_2][1];
>> + const u8 r10 = bpf2a32[TMP_REG_2][0];
>> + const u8 fplo = bpf2a32[BPF_REG_FP][1];
>> + const u8 fphi = bpf2a32[BPF_REG_FP][0];
>> + const u8 sp = ARM_SP;
>> + const u8 *tcc = bpf2a32[TCALL_CNT];
>> +
>> + u16 reg_set = 0;
>>
>> - for (i = 0; i < prog->len; i++) {
>> - u16 code;
>> + /*
>> + * eBPF prog stack layout
>> + *
>> + * high
>> + * original ARM_SP => +-----+ eBPF prologue
>> + * |FP/LR|
>> + * current ARM_FP => +-----+
>> + * | ... | callee saved registers
>> + * eBPF fp register => +-----+ <= (BPF_FP)
>> + * | ... | eBPF JIT scratch space
>> + * | | eBPF prog stack
>> + * +-----+
>> + * |RSVD | JIT scratchpad
>> + * current A64_SP => +-----+ <= (BPF_FP - STACK_SIZE)
>> + * | |
>> + * | ... | Function call stack
>> + * | |
>> + * +-----+
>> + * low
>> + */
>>
>> - inst = &(prog->insns[i]);
>> - /* K as an immediate value operand */
>> - k = inst->k;
>> - code = bpf_anc_helper(inst);
>> + /* Save callee saved registers. */
>> + reg_set |= (1<<r4) | (1<<r5) | (1<<r6) | (1<<r7) | (1<<r8) | (1<<r10);
>> +#ifdef CONFIG_FRAME_POINTER
>> + reg_set |= (1<<ARM_FP) | (1<<ARM_IP) | (1<<ARM_LR) | (1<<ARM_PC);
>> + emit(ARM_MOV_R(ARM_IP, sp), ctx);
>> + emit(ARM_PUSH(reg_set), ctx);
>> + emit(ARM_SUB_I(ARM_FP, ARM_IP, 4), ctx);
>> +#else
>> + /* Check if call instruction exists in BPF body */
>> + if (ctx->seen & SEEN_CALL)
>> + reg_set |= (1<<ARM_LR);
>> + emit(ARM_PUSH(reg_set), ctx);
>> +#endif
>> + /* Save frame pointer for later */
>> + emit(ARM_SUB_I(ARM_IP, sp, SCRATCH_SIZE), ctx);
>> +
>> + /* Set up function call stack */
>> + emit(ARM_SUB_I(ARM_SP, ARM_SP, imm8m(STACK_SIZE)), ctx);
>> +
>> + /* Set up BPF prog stack base register */
>> + emit_a32_mov_r(fplo, ARM_IP, true, false, ctx);
>> + emit_a32_mov_i(fphi, 0, true, ctx);
>> +
>> + /* mov r4, 0 */
>> + emit(ARM_MOV_I(r4, 0), ctx);
>> + /* MOV bpf_ctx pointer to BPF_R1 */
>> + emit(ARM_MOV_R(r3, r4), ctx);
>> + emit(ARM_MOV_R(r2, r0), ctx);
>> + /* Initialize Tail Count */
>> + emit(ARM_STR_I(r4, ARM_SP, STACK_VAR(tcc[0])), ctx);
>> + emit(ARM_STR_I(r4, ARM_SP, STACK_VAR(tcc[1])), ctx);
>> + /* end of prologue */
>> +}
>>
>> - /* compute offsets only in the fake pass */
>> - if (ctx->target == NULL)
>> - ctx->offsets[i] = ctx->idx * 4;
>> +static void build_epilogue(struct jit_ctx *ctx)
>> +{
>> + const u8 r4 = bpf2a32[BPF_REG_6][1];
>> + const u8 r5 = bpf2a32[BPF_REG_6][0];
>> + const u8 r6 = bpf2a32[TMP_REG_1][1];
>> + const u8 r7 = bpf2a32[TMP_REG_1][0];
>> + const u8 r8 = bpf2a32[TMP_REG_2][1];
>> + const u8 r10 = bpf2a32[TMP_REG_2][0];
>> + u16 reg_set = 0;
>> +
>> + /* unwind function call stack */
>> + emit(ARM_ADD_I(ARM_SP, ARM_SP, imm8m(STACK_SIZE)), ctx);
>> +
>> + /* restore callee saved registers. */
>> + reg_set |= (1<<r4) | (1<<r5) | (1<<r6) | (1<<r7) | (1<<r8) | (1<<r10);
>> +#ifdef CONFIG_FRAME_POINTER
>> + /* the first instruction of the prologue was: mov ip, sp */
>> + reg_set |= (1<<ARM_FP) | (1<<ARM_SP) | (1<<ARM_PC);
>> + emit(ARM_LDM(ARM_SP, reg_set), ctx);
>> +#else
>> + if (ctx->seen & SEEN_CALL)
>> + reg_set |= (1<<ARM_PC);
>> + /* Restore callee saved registers. */
>> + emit(ARM_POP(reg_set), ctx);
>> + /* Return back to the callee function */
>> + if (!(ctx->seen & SEEN_CALL))
>> + emit(ARM_BX(ARM_LR), ctx);
>> +#endif
>> +}
>>
>> - switch (code) {
>> - case BPF_LD | BPF_IMM:
>> - emit_mov_i(r_A, k, ctx);
>> +/*
>> + * Convert an eBPF instruction to native instruction, i.e
>> + * JITs an eBPF instruction.
>> + * Returns :
>> + * 0 - Successfully JITed an 8-byte eBPF instruction
>> + * >0 - Successfully JITed a 16-byte eBPF instruction
>> + * <0 - Failed to JIT.
>> + */
>> +static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx)
>> +{
>> + const u8 code = insn->code;
>> + const u8 *dst = bpf2a32[insn->dst_reg];
>> + const u8 *src = bpf2a32[insn->src_reg];
>> + const u8 *tmp = bpf2a32[TMP_REG_1];
>> + const u8 *tmp2 = bpf2a32[TMP_REG_2];
>> + const s16 off = insn->off;
>> + const s32 imm = insn->imm;
>> + const int i = insn - ctx->prog->insnsi;
>> + const bool is64 = BPF_CLASS(code) == BPF_ALU64;
>> + const bool dstk = is_on_stack(insn->dst_reg);
>> + const bool sstk = is_on_stack(insn->src_reg);
>> + u8 rd, rt, rm, rn;
>> + s32 jmp_offset;
>> +
>> +#define check_imm(bits, imm) do { \
>> + if ((((imm) > 0) && ((imm) >> (bits))) || \
>> + (((imm) < 0) && (~(imm) >> (bits)))) { \
>> + pr_info("[%2d] imm=%d(0x%x) out of range\n", \
>> + i, imm, imm); \
>> + return -EINVAL; \
>> + } \
>> +} while (0)
>> +#define check_imm24(imm) check_imm(24, imm)
>> +
>> + switch (code) {
>> + /* ALU operations */
>> +
>> + /* dst = src */
>> + case BPF_ALU | BPF_MOV | BPF_K:
>> + case BPF_ALU | BPF_MOV | BPF_X:
>> + case BPF_ALU64 | BPF_MOV | BPF_K:
>> + case BPF_ALU64 | BPF_MOV | BPF_X:
>> + switch (BPF_SRC(code)) {
>> + case BPF_X:
>> + emit_a32_mov_r64(is64, dst, src, dstk, sstk, ctx);
>> break;
>> - case BPF_LD | BPF_W | BPF_LEN:
>> - ctx->seen |= SEEN_SKB;
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, len) != 4);
>> - emit(ARM_LDR_I(r_A, r_skb,
>> - offsetof(struct sk_buff, len)), ctx);
>> + case BPF_K:
>> + /* Sign-extend immediate value to destination reg */
>> + emit_a32_mov_i64(is64, dst, imm, dstk, ctx);
>> break;
>> - case BPF_LD | BPF_MEM:
>> - /* A = scratch[k] */
>> - ctx->seen |= SEEN_MEM_WORD(k);
>> - emit(ARM_LDR_I(r_A, ARM_SP, SCRATCH_OFF(k)), ctx);
>> + }
>> + break;
>> + /* dst = dst + src/imm */
>> + /* dst = dst - src/imm */
>> + /* dst = dst | src/imm */
>> + /* dst = dst & src/imm */
>> + /* dst = dst ^ src/imm */
>> + /* dst = dst * src/imm */
>> + /* dst = dst << src */
>> + /* dst = dst >> src */
>> + case BPF_ALU | BPF_ADD | BPF_K:
>> + case BPF_ALU | BPF_ADD | BPF_X:
>> + case BPF_ALU | BPF_SUB | BPF_K:
>> + case BPF_ALU | BPF_SUB | BPF_X:
>> + case BPF_ALU | BPF_OR | BPF_K:
>> + case BPF_ALU | BPF_OR | BPF_X:
>> + case BPF_ALU | BPF_AND | BPF_K:
>> + case BPF_ALU | BPF_AND | BPF_X:
>> + case BPF_ALU | BPF_XOR | BPF_K:
>> + case BPF_ALU | BPF_XOR | BPF_X:
>> + case BPF_ALU | BPF_MUL | BPF_K:
>> + case BPF_ALU | BPF_MUL | BPF_X:
>> + case BPF_ALU | BPF_LSH | BPF_X:
>> + case BPF_ALU | BPF_RSH | BPF_X:
>> + case BPF_ALU | BPF_ARSH | BPF_K:
>> + case BPF_ALU | BPF_ARSH | BPF_X:
>> + case BPF_ALU64 | BPF_ADD | BPF_K:
>> + case BPF_ALU64 | BPF_ADD | BPF_X:
>> + case BPF_ALU64 | BPF_SUB | BPF_K:
>> + case BPF_ALU64 | BPF_SUB | BPF_X:
>> + case BPF_ALU64 | BPF_OR | BPF_K:
>> + case BPF_ALU64 | BPF_OR | BPF_X:
>> + case BPF_ALU64 | BPF_AND | BPF_K:
>> + case BPF_ALU64 | BPF_AND | BPF_X:
>> + case BPF_ALU64 | BPF_XOR | BPF_K:
>> + case BPF_ALU64 | BPF_XOR | BPF_X:
>> + switch (BPF_SRC(code)) {
>> + case BPF_X:
>> + emit_a32_alu_r64(is64, dst, src, dstk, sstk,
>> + ctx, BPF_OP(code));
>> break;
>> - case BPF_LD | BPF_W | BPF_ABS:
>> - load_order = 2;
>> - goto load;
>> - case BPF_LD | BPF_H | BPF_ABS:
>> - load_order = 1;
>> - goto load;
>> - case BPF_LD | BPF_B | BPF_ABS:
>> - load_order = 0;
>> -load:
>> - emit_mov_i(r_off, k, ctx);
>> -load_common:
>> - ctx->seen |= SEEN_DATA | SEEN_CALL;
>> -
>> - if (load_order > 0) {
>> - emit(ARM_SUB_I(r_scratch, r_skb_hl,
>> - 1 << load_order), ctx);
>> - emit(ARM_CMP_R(r_scratch, r_off), ctx);
>> - condt = ARM_COND_GE;
>> - } else {
>> - emit(ARM_CMP_R(r_skb_hl, r_off), ctx);
>> - condt = ARM_COND_HI;
>> - }
>> -
>> - /*
>> - * test for negative offset, only if we are
>> - * currently scheduled to take the fast
>> - * path. this will update the flags so that
>> - * the slowpath instruction are ignored if the
>> - * offset is negative.
>> - *
>> - * for loard_order == 0 the HI condition will
>> - * make loads at offset 0 take the slow path too.
>> + case BPF_K:
>> + /* Move immediate value to the temporary register
>> + * and then do the ALU operation on the temporary
>> + * register as this will sign-extend the immediate
>> + * value into temporary reg and then it would be
>> + * safe to do the operation on it.
>> */
>> - _emit(condt, ARM_CMP_I(r_off, 0), ctx);
>> -
>> - _emit(condt, ARM_ADD_R(r_scratch, r_off, r_skb_data),
>> - ctx);
>> -
>> - if (load_order == 0)
>> - _emit(condt, ARM_LDRB_I(r_A, r_scratch, 0),
>> - ctx);
>> - else if (load_order == 1)
>> - emit_load_be16(condt, r_A, r_scratch, ctx);
>> - else if (load_order == 2)
>> - emit_load_be32(condt, r_A, r_scratch, ctx);
>> -
>> - _emit(condt, ARM_B(b_imm(i + 1, ctx)), ctx);
>> -
>> - /* the slowpath */
>> - emit_mov_i(ARM_R3, (u32)load_func[load_order], ctx);
>> - emit(ARM_MOV_R(ARM_R0, r_skb), ctx);
>> - /* the offset is already in R1 */
>> - emit_blx_r(ARM_R3, ctx);
>> - /* check the result of skb_copy_bits */
>> - emit(ARM_CMP_I(ARM_R1, 0), ctx);
>> - emit_err_ret(ARM_COND_NE, ctx);
>> - emit(ARM_MOV_R(r_A, ARM_R0), ctx);
>> + emit_a32_mov_i64(is64, tmp2, imm, false, ctx);
>> + emit_a32_alu_r64(is64, dst, tmp2, dstk, false,
>> + ctx, BPF_OP(code));
>> break;
>> - case BPF_LD | BPF_W | BPF_IND:
>> - load_order = 2;
>> - goto load_ind;
>> - case BPF_LD | BPF_H | BPF_IND:
>> - load_order = 1;
>> - goto load_ind;
>> - case BPF_LD | BPF_B | BPF_IND:
>> - load_order = 0;
>> -load_ind:
>> - update_on_xread(ctx);
>> - OP_IMM3(ARM_ADD, r_off, r_X, k, ctx);
>> - goto load_common;
>> - case BPF_LDX | BPF_IMM:
>> - ctx->seen |= SEEN_X;
>> - emit_mov_i(r_X, k, ctx);
>> + }
>> + break;
>> + /* dst = dst / src(imm) */
>> + /* dst = dst % src(imm) */
>> + case BPF_ALU | BPF_DIV | BPF_K:
>> + case BPF_ALU | BPF_DIV | BPF_X:
>> + case BPF_ALU | BPF_MOD | BPF_K:
>> + case BPF_ALU | BPF_MOD | BPF_X:
>> + rt = src_lo;
>> + rd = dstk ? tmp2[1] : dst_lo;
>> + if (dstk)
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + switch (BPF_SRC(code)) {
>> + case BPF_X:
>> + rt = sstk ? tmp2[0] : rt;
>> + if (sstk)
>> + emit(ARM_LDR_I(rt, ARM_SP, STACK_VAR(src_lo)),
>> + ctx);
>> break;
>> - case BPF_LDX | BPF_W | BPF_LEN:
>> - ctx->seen |= SEEN_X | SEEN_SKB;
>> - emit(ARM_LDR_I(r_X, r_skb,
>> - offsetof(struct sk_buff, len)), ctx);
>> + case BPF_K:
>> + rt = tmp2[0];
>> + emit_a32_mov_i(rt, imm, false, ctx);
>> break;
>> - case BPF_LDX | BPF_MEM:
>> - ctx->seen |= SEEN_X | SEEN_MEM_WORD(k);
>> - emit(ARM_LDR_I(r_X, ARM_SP, SCRATCH_OFF(k)), ctx);
>> + }
>> + emit_udivmod(rd, rd, rt, ctx, BPF_OP(code));
>> + if (dstk)
>> + emit(ARM_STR_I(rd, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit_a32_mov_i(dst_hi, 0, dstk, ctx);
>> + break;
>> + case BPF_ALU64 | BPF_DIV | BPF_K:
>> + case BPF_ALU64 | BPF_DIV | BPF_X:
>> + case BPF_ALU64 | BPF_MOD | BPF_K:
>> + case BPF_ALU64 | BPF_MOD | BPF_X:
>> + goto notyet;
>> + /* dst = dst >> imm */
>> + /* dst = dst << imm */
>> + case BPF_ALU | BPF_RSH | BPF_K:
>> + case BPF_ALU | BPF_LSH | BPF_K:
>> + if (unlikely(imm > 31))
>> + return -EINVAL;
>> + if (imm)
>> + emit_a32_alu_i(dst_lo, imm, dstk, ctx, BPF_OP(code));
>> + emit_a32_mov_i(dst_hi, 0, dstk, ctx);
>> + break;
>> + /* dst = dst << imm */
>> + case BPF_ALU64 | BPF_LSH | BPF_K:
>> + if (unlikely(imm > 63))
>> + return -EINVAL;
>> + emit_a32_lsh_i64(dst, dstk, imm, ctx);
>> + break;
>> + /* dst = dst >> imm */
>> + case BPF_ALU64 | BPF_RSH | BPF_K:
>> + if (unlikely(imm > 63))
>> + return -EINVAL;
>> + emit_a32_lsr_i64(dst, dstk, imm, ctx);
>> + break;
>> + /* dst = dst << src */
>> + case BPF_ALU64 | BPF_LSH | BPF_X:
>> + emit_a32_lsh_r64(dst, src, dstk, sstk, ctx);
>> + break;
>> + /* dst = dst >> src */
>> + case BPF_ALU64 | BPF_RSH | BPF_X:
>> + emit_a32_lsr_r64(dst, src, dstk, sstk, ctx);
>> + break;
>> + /* dst = dst >> src (signed) */
>> + case BPF_ALU64 | BPF_ARSH | BPF_X:
>> + emit_a32_arsh_r64(dst, src, dstk, sstk, ctx);
>> + break;
>> + /* dst = dst >> imm (signed) */
>> + case BPF_ALU64 | BPF_ARSH | BPF_K:
>> + if (unlikely(imm > 63))
>> + return -EINVAL;
>> + emit_a32_arsh_i64(dst, dstk, imm, ctx);
>> + break;
>> + /* dst = ~dst */
>> + case BPF_ALU | BPF_NEG:
>> + emit_a32_alu_i(dst_lo, 0, dstk, ctx, BPF_OP(code));
>> + emit_a32_mov_i(dst_hi, 0, dstk, ctx);
>> + break;
>> + /* dst = ~dst (64 bit) */
>> + case BPF_ALU64 | BPF_NEG:
>> + emit_a32_neg64(dst, dstk, ctx);
>> + break;
>> + /* dst = dst * src/imm */
>> + case BPF_ALU64 | BPF_MUL | BPF_X:
>> + case BPF_ALU64 | BPF_MUL | BPF_K:
>> + switch (BPF_SRC(code)) {
>> + case BPF_X:
>> + emit_a32_mul_r64(dst, src, dstk, sstk, ctx);
>> break;
>> - case BPF_LDX | BPF_B | BPF_MSH:
>> - /* x = ((*(frame + k)) & 0xf) << 2; */
>> - ctx->seen |= SEEN_X | SEEN_DATA | SEEN_CALL;
>> - /* the interpreter should deal with the negative K */
>> - if ((int)k < 0)
>> - return -1;
>> - /* offset in r1: we might have to take the slow path */
>> - emit_mov_i(r_off, k, ctx);
>> - emit(ARM_CMP_R(r_skb_hl, r_off), ctx);
>> -
>> - /* load in r0: common with the slowpath */
>> - _emit(ARM_COND_HI, ARM_LDRB_R(ARM_R0, r_skb_data,
>> - ARM_R1), ctx);
>> - /*
>> - * emit_mov_i() might generate one or two instructions,
>> - * the same holds for emit_blx_r()
>> + case BPF_K:
>> + /* Move immediate value to the temporary register
>> + * and then do the multiplication on it as this
>> + * will sign-extend the immediate value into temp
>> + * reg then it would be safe to do the operation
>> + * on it.
>> */
>> - _emit(ARM_COND_HI, ARM_B(b_imm(i + 1, ctx) - 2), ctx);
>> -
>> - emit(ARM_MOV_R(ARM_R0, r_skb), ctx);
>> - /* r_off is r1 */
>> - emit_mov_i(ARM_R3, (u32)jit_get_skb_b, ctx);
>> - emit_blx_r(ARM_R3, ctx);
>> - /* check the return value of skb_copy_bits */
>> - emit(ARM_CMP_I(ARM_R1, 0), ctx);
>> - emit_err_ret(ARM_COND_NE, ctx);
>> -
>> - emit(ARM_AND_I(r_X, ARM_R0, 0x00f), ctx);
>> - emit(ARM_LSL_I(r_X, r_X, 2), ctx);
>> - break;
>> - case BPF_ST:
>> - ctx->seen |= SEEN_MEM_WORD(k);
>> - emit(ARM_STR_I(r_A, ARM_SP, SCRATCH_OFF(k)), ctx);
>> - break;
>> - case BPF_STX:
>> - update_on_xread(ctx);
>> - ctx->seen |= SEEN_MEM_WORD(k);
>> - emit(ARM_STR_I(r_X, ARM_SP, SCRATCH_OFF(k)), ctx);
>> - break;
>> - case BPF_ALU | BPF_ADD | BPF_K:
>> - /* A += K */
>> - OP_IMM3(ARM_ADD, r_A, r_A, k, ctx);
>> - break;
>> - case BPF_ALU | BPF_ADD | BPF_X:
>> - update_on_xread(ctx);
>> - emit(ARM_ADD_R(r_A, r_A, r_X), ctx);
>> - break;
>> - case BPF_ALU | BPF_SUB | BPF_K:
>> - /* A -= K */
>> - OP_IMM3(ARM_SUB, r_A, r_A, k, ctx);
>> - break;
>> - case BPF_ALU | BPF_SUB | BPF_X:
>> - update_on_xread(ctx);
>> - emit(ARM_SUB_R(r_A, r_A, r_X), ctx);
>> - break;
>> - case BPF_ALU | BPF_MUL | BPF_K:
>> - /* A *= K */
>> - emit_mov_i(r_scratch, k, ctx);
>> - emit(ARM_MUL(r_A, r_A, r_scratch), ctx);
>> - break;
>> - case BPF_ALU | BPF_MUL | BPF_X:
>> - update_on_xread(ctx);
>> - emit(ARM_MUL(r_A, r_A, r_X), ctx);
>> - break;
>> - case BPF_ALU | BPF_DIV | BPF_K:
>> - if (k == 1)
>> - break;
>> - emit_mov_i(r_scratch, k, ctx);
>> - emit_udivmod(r_A, r_A, r_scratch, ctx, BPF_DIV);
>> - break;
>> - case BPF_ALU | BPF_DIV | BPF_X:
>> - update_on_xread(ctx);
>> - emit(ARM_CMP_I(r_X, 0), ctx);
>> - emit_err_ret(ARM_COND_EQ, ctx);
>> - emit_udivmod(r_A, r_A, r_X, ctx, BPF_DIV);
>> - break;
>> - case BPF_ALU | BPF_MOD | BPF_K:
>> - if (k == 1) {
>> - emit_mov_i(r_A, 0, ctx);
>> - break;
>> - }
>> - emit_mov_i(r_scratch, k, ctx);
>> - emit_udivmod(r_A, r_A, r_scratch, ctx, BPF_MOD);
>> + emit_a32_mov_i64(is64, tmp2, imm, false, ctx);
>> + emit_a32_mul_r64(dst, tmp2, dstk, false, ctx);
>> break;
>> - case BPF_ALU | BPF_MOD | BPF_X:
>> - update_on_xread(ctx);
>> - emit(ARM_CMP_I(r_X, 0), ctx);
>> - emit_err_ret(ARM_COND_EQ, ctx);
>> - emit_udivmod(r_A, r_A, r_X, ctx, BPF_MOD);
>> - break;
>> - case BPF_ALU | BPF_OR | BPF_K:
>> - /* A |= K */
>> - OP_IMM3(ARM_ORR, r_A, r_A, k, ctx);
>> + }
>> + break;
>> + /* dst = htole(dst) */
>> + /* dst = htobe(dst) */
>> + case BPF_ALU | BPF_END | BPF_FROM_LE:
>> + case BPF_ALU | BPF_END | BPF_FROM_BE:
>> + rd = dstk ? tmp[0] : dst_hi;
>> + rt = dstk ? tmp[1] : dst_lo;
>> + if (dstk) {
>> + emit(ARM_LDR_I(rt, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> + if (BPF_SRC(code) == BPF_FROM_LE)
>> + goto emit_bswap_uxt;
>> + switch (imm) {
>> + case 16:
>> + emit_rev16(rt, rt, ctx);
>> + goto emit_bswap_uxt;
>> + case 32:
>> + emit_rev32(rt, rt, ctx);
>> + goto emit_bswap_uxt;
>> + case 64:
>> + /* Because of the usage of ARM_LR */
>> + ctx->seen |= SEEN_CALL;
>> + emit_rev32(ARM_LR, rt, ctx);
>> + emit_rev32(rt, rd, ctx);
>> + emit(ARM_MOV_R(rd, ARM_LR), ctx);
>> break;
>> - case BPF_ALU | BPF_OR | BPF_X:
>> - update_on_xread(ctx);
>> - emit(ARM_ORR_R(r_A, r_A, r_X), ctx);
>> + }
>> + goto exit;
>> +emit_bswap_uxt:
>> + switch (imm) {
>> + case 16:
>> + /* zero-extend 16 bits into 64 bits */
>> +#if __LINUX_ARM_ARCH__ < 6
>> + emit_a32_mov_i(tmp2[1], 0xffff, false, ctx);
>> + emit(ARM_AND_R(rt, rt, tmp2[1]), ctx);
>> +#else /* ARMv6+ */
>> + emit(ARM_UXTH(rt, rt), ctx);
>> +#endif
>> + emit(ARM_EOR_R(rd, rd, rd), ctx);
>> break;
>> - case BPF_ALU | BPF_XOR | BPF_K:
>> - /* A ^= K; */
>> - OP_IMM3(ARM_EOR, r_A, r_A, k, ctx);
>> + case 32:
>> + /* zero-extend 32 bits into 64 bits */
>> + emit(ARM_EOR_R(rd, rd, rd), ctx);
>> break;
>> - case BPF_ANC | SKF_AD_ALU_XOR_X:
>> - case BPF_ALU | BPF_XOR | BPF_X:
>> - /* A ^= X */
>> - update_on_xread(ctx);
>> - emit(ARM_EOR_R(r_A, r_A, r_X), ctx);
>> + case 64:
>> + /* nop */
>> break;
>> - case BPF_ALU | BPF_AND | BPF_K:
>> - /* A &= K */
>> - OP_IMM3(ARM_AND, r_A, r_A, k, ctx);
>> + }
>> +exit:
>> + if (dstk) {
>> + emit(ARM_STR_I(rt, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_STR_I(rd, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> + break;
>> + /* dst = imm64 */
>> + case BPF_LD | BPF_IMM | BPF_DW:
>> + {
>> + const struct bpf_insn insn1 = insn[1];
>> + u32 hi, lo = imm;
>> +
>> + if (insn1.code != 0 || insn1.src_reg != 0 ||
>> + insn1.dst_reg != 0 || insn1.off != 0) {
>> + /* Note: verifier in BPF core must catch invalid
>> + * instruction.
>> + */
>> + pr_err_once("Invalid BPF_LD_IMM64 instruction\n");
>> + return -EINVAL;
>> + }
>> + hi = insn1.imm;
>> + emit_a32_mov_i(dst_lo, lo, dstk, ctx);
>> + emit_a32_mov_i(dst_hi, hi, dstk, ctx);
>> +
>> + return 1;
>> + }
>> + /* LDX: dst = *(size *)(src + off) */
>> + case BPF_LDX | BPF_MEM | BPF_W:
>> + case BPF_LDX | BPF_MEM | BPF_H:
>> + case BPF_LDX | BPF_MEM | BPF_B:
>> + case BPF_LDX | BPF_MEM | BPF_DW:
>> + rn = sstk ? tmp2[1] : src_lo;
>> + if (sstk)
>> + emit(ARM_LDR_I(rn, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + switch (BPF_SIZE(code)) {
>> + case BPF_W:
>> + /* Load a Word */
>> + case BPF_H:
>> + /* Load a Half-Word */
>> + case BPF_B:
>> + /* Load a Byte */
>> + emit_ldx_r(dst_lo, rn, dstk, off, ctx, BPF_SIZE(code));
>> + emit_a32_mov_i(dst_hi, 0, dstk, ctx);
>> break;
>> - case BPF_ALU | BPF_AND | BPF_X:
>> - update_on_xread(ctx);
>> - emit(ARM_AND_R(r_A, r_A, r_X), ctx);
>> + case BPF_DW:
>> + /* Load a double word */
>> + emit_ldx_r(dst_lo, rn, dstk, off, ctx, BPF_W);
>> + emit_ldx_r(dst_hi, rn, dstk, off+4, ctx, BPF_W);
>> break;
>> - case BPF_ALU | BPF_LSH | BPF_K:
>> - if (unlikely(k > 31))
>> - return -1;
>> - emit(ARM_LSL_I(r_A, r_A, k), ctx);
>> + }
>> + break;
>> + /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + imm)) */
>> + case BPF_LD | BPF_ABS | BPF_W:
>> + case BPF_LD | BPF_ABS | BPF_H:
>> + case BPF_LD | BPF_ABS | BPF_B:
>> + /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + src + imm)) */
>> + case BPF_LD | BPF_IND | BPF_W:
>> + case BPF_LD | BPF_IND | BPF_H:
>> + case BPF_LD | BPF_IND | BPF_B:
>> + {
>> + const u8 r4 = bpf2a32[BPF_REG_6][1]; /* r4 = ptr to sk_buff */
>> + const u8 r0 = bpf2a32[BPF_REG_0][1]; /*r0: struct sk_buff *skb*/
>> + /* rtn value */
>> + const u8 r1 = bpf2a32[BPF_REG_0][0]; /* r1: int k */
>> + const u8 r2 = bpf2a32[BPF_REG_1][1]; /* r2: unsigned int size */
>> + const u8 r3 = bpf2a32[BPF_REG_1][0]; /* r3: void *buffer */
>> + const u8 r6 = bpf2a32[TMP_REG_1][1]; /* r6: void *(*func)(..) */
>> + int size;
>> +
>> + /* Setting up first argument */
>> + emit(ARM_MOV_R(r0, r4), ctx);
>> +
>> + /* Setting up second argument */
>> + emit_a32_mov_i(r1, imm, false, ctx);
>> + if (BPF_MODE(code) == BPF_IND)
>> + emit_a32_alu_r(r1, src_lo, false, sstk, ctx,
>> + false, false, BPF_ADD);
>> +
>> + /* Setting up third argument */
>> + switch (BPF_SIZE(code)) {
>> + case BPF_W:
>> + size = 4;
>> break;
>> - case BPF_ALU | BPF_LSH | BPF_X:
>> - update_on_xread(ctx);
>> - emit(ARM_LSL_R(r_A, r_A, r_X), ctx);
>> + case BPF_H:
>> + size = 2;
>> break;
>> - case BPF_ALU | BPF_RSH | BPF_K:
>> - if (unlikely(k > 31))
>> - return -1;
>> - if (k)
>> - emit(ARM_LSR_I(r_A, r_A, k), ctx);
>> + case BPF_B:
>> + size = 1;
>> break;
>> - case BPF_ALU | BPF_RSH | BPF_X:
>> - update_on_xread(ctx);
>> - emit(ARM_LSR_R(r_A, r_A, r_X), ctx);
>> + default:
>> + return -EINVAL;
>> + }
>> + emit_a32_mov_i(r2, size, false, ctx);
>> +
>> + /* Setting up fourth argument */
>> + emit(ARM_ADD_I(r3, ARM_SP, imm8m(SKB_BUFFER)), ctx);
>> +
>> + /* Setting up function pointer to call */
>> + emit_a32_mov_i(r6, (unsigned int)bpf_load_pointer, false, ctx);
>> + emit_blx_r(r6, ctx);
>> +
>> + emit(ARM_EOR_R(r1, r1, r1), ctx);
>> + /* Check if return address is NULL or not.
>> + * if NULL then jump to epilogue
>> + * else continue to load the value from retn address
>> + */
>> + emit(ARM_CMP_I(r0, 0), ctx);
>> + jmp_offset = epilogue_offset(ctx);
>> + check_imm24(jmp_offset);
>> + _emit(ARM_COND_EQ, ARM_B(jmp_offset), ctx);
>> +
>> + /* Load value from the address */
>> + switch (BPF_SIZE(code)) {
>> + case BPF_W:
>> + emit(ARM_LDR_I(r0, r0, 0), ctx);
>> + emit_rev32(r0, r0, ctx);
>> break;
>> - case BPF_ALU | BPF_NEG:
>> - /* A = -A */
>> - emit(ARM_RSB_I(r_A, r_A, 0), ctx);
>> + case BPF_H:
>> + emit(ARM_LDRH_I(r0, r0, 0), ctx);
>> + emit_rev16(r0, r0, ctx);
>> break;
>> - case BPF_JMP | BPF_JA:
>> - /* pc += K */
>> - emit(ARM_B(b_imm(i + k + 1, ctx)), ctx);
>> + case BPF_B:
>> + emit(ARM_LDRB_I(r0, r0, 0), ctx);
>> + /* No need to reverse */
>> break;
>> - case BPF_JMP | BPF_JEQ | BPF_K:
>> - /* pc += (A == K) ? pc->jt : pc->jf */
>> - condt = ARM_COND_EQ;
>> - goto cmp_imm;
>> - case BPF_JMP | BPF_JGT | BPF_K:
>> - /* pc += (A > K) ? pc->jt : pc->jf */
>> - condt = ARM_COND_HI;
>> - goto cmp_imm;
>> - case BPF_JMP | BPF_JGE | BPF_K:
>> - /* pc += (A >= K) ? pc->jt : pc->jf */
>> - condt = ARM_COND_HS;
>> -cmp_imm:
>> - imm12 = imm8m(k);
>> - if (imm12 < 0) {
>> - emit_mov_i_no8m(r_scratch, k, ctx);
>> - emit(ARM_CMP_R(r_A, r_scratch), ctx);
>> - } else {
>> - emit(ARM_CMP_I(r_A, imm12), ctx);
>> - }
>> -cond_jump:
>> - if (inst->jt)
>> - _emit(condt, ARM_B(b_imm(i + inst->jt + 1,
>> - ctx)), ctx);
>> - if (inst->jf)
>> - _emit(condt ^ 1, ARM_B(b_imm(i + inst->jf + 1,
>> - ctx)), ctx);
>> + }
>> + break;
>> + }
>> + /* ST: *(size *)(dst + off) = imm */
>> + case BPF_ST | BPF_MEM | BPF_W:
>> + case BPF_ST | BPF_MEM | BPF_H:
>> + case BPF_ST | BPF_MEM | BPF_B:
>> + case BPF_ST | BPF_MEM | BPF_DW:
>> + switch (BPF_SIZE(code)) {
>> + case BPF_DW:
>> + /* Sign-extend immediate value into temp reg */
>> + emit_a32_mov_i64(true, tmp2, imm, false, ctx);
>> + emit_str_r(dst_lo, tmp2[1], dstk, off, ctx, BPF_W);
>> + emit_str_r(dst_lo, tmp2[0], dstk, off+4, ctx, BPF_W);
>> break;
>> - case BPF_JMP | BPF_JEQ | BPF_X:
>> - /* pc += (A == X) ? pc->jt : pc->jf */
>> - condt = ARM_COND_EQ;
>> - goto cmp_x;
>> - case BPF_JMP | BPF_JGT | BPF_X:
>> - /* pc += (A > X) ? pc->jt : pc->jf */
>> - condt = ARM_COND_HI;
>> - goto cmp_x;
>> - case BPF_JMP | BPF_JGE | BPF_X:
>> - /* pc += (A >= X) ? pc->jt : pc->jf */
>> - condt = ARM_COND_CS;
>> -cmp_x:
>> - update_on_xread(ctx);
>> - emit(ARM_CMP_R(r_A, r_X), ctx);
>> - goto cond_jump;
>> - case BPF_JMP | BPF_JSET | BPF_K:
>> - /* pc += (A & K) ? pc->jt : pc->jf */
>> - condt = ARM_COND_NE;
>> - /* not set iff all zeroes iff Z==1 iff EQ */
>> -
>> - imm12 = imm8m(k);
>> - if (imm12 < 0) {
>> - emit_mov_i_no8m(r_scratch, k, ctx);
>> - emit(ARM_TST_R(r_A, r_scratch), ctx);
>> - } else {
>> - emit(ARM_TST_I(r_A, imm12), ctx);
>> - }
>> - goto cond_jump;
>> - case BPF_JMP | BPF_JSET | BPF_X:
>> - /* pc += (A & X) ? pc->jt : pc->jf */
>> - update_on_xread(ctx);
>> - condt = ARM_COND_NE;
>> - emit(ARM_TST_R(r_A, r_X), ctx);
>> - goto cond_jump;
>> - case BPF_RET | BPF_A:
>> - emit(ARM_MOV_R(ARM_R0, r_A), ctx);
>> - goto b_epilogue;
>> - case BPF_RET | BPF_K:
>> - if ((k == 0) && (ctx->ret0_fp_idx < 0))
>> - ctx->ret0_fp_idx = i;
>> - emit_mov_i(ARM_R0, k, ctx);
>> -b_epilogue:
>> - if (i != ctx->skf->len - 1)
>> - emit(ARM_B(b_imm(prog->len, ctx)), ctx);
>> + case BPF_W:
>> + case BPF_H:
>> + case BPF_B:
>> + emit_a32_mov_i(tmp2[1], imm, false, ctx);
>> + emit_str_r(dst_lo, tmp2[1], dstk, off, ctx,
>> + BPF_SIZE(code));
>> break;
>> - case BPF_MISC | BPF_TAX:
>> - /* X = A */
>> - ctx->seen |= SEEN_X;
>> - emit(ARM_MOV_R(r_X, r_A), ctx);
>> + }
>> + break;
>> + /* STX XADD: lock *(u32 *)(dst + off) += src */
>> + case BPF_STX | BPF_XADD | BPF_W:
>> + /* STX XADD: lock *(u64 *)(dst + off) += src */
>> + case BPF_STX | BPF_XADD | BPF_DW:
>> + goto notyet;
>> + /* STX: *(size *)(dst + off) = src */
>> + case BPF_STX | BPF_MEM | BPF_W:
>> + case BPF_STX | BPF_MEM | BPF_H:
>> + case BPF_STX | BPF_MEM | BPF_B:
>> + case BPF_STX | BPF_MEM | BPF_DW:
>> + {
>> + u8 sz = BPF_SIZE(code);
>> +
>> + rn = sstk ? tmp2[1] : src_lo;
>> + rm = sstk ? tmp2[0] : src_hi;
>> + if (!sstk)
>> + goto do_store;
>> + switch (BPF_SIZE(code)) {
>> + case BPF_W:
>> + emit(ARM_LDR_I(rn, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + goto empty_hi;
>> + case BPF_H:
>> + emit(ARM_LDRH_I(rn, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + goto empty_hi;
>> + case BPF_B:
>> + emit(ARM_LDRB_I(rn, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + goto empty_hi;
>> +empty_hi:
>> + emit(ARM_EOR_R(rm, rm, rm), ctx);
>> + case BPF_DW:
>> + emit(ARM_LDR_I(rn, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(src_hi)), ctx);
>> + sz = BPF_W;
>> break;
>> - case BPF_MISC | BPF_TXA:
>> - /* A = X */
>> - update_on_xread(ctx);
>> - emit(ARM_MOV_R(r_A, r_X), ctx);
>> + }
>> +
>> +do_store:
>> + /* Clear higher word except for BPF_DW */
>> + if (BPF_SIZE(code) != BPF_DW)
>> + emit(ARM_EOR_R(rm, rm, rm), ctx);
>> +
>> + /* Store the value */
>> + emit_str_r(dst_lo, rn, dstk, off, ctx, sz);
>> + emit_str_r(dst_lo, rm, dstk, off+4, ctx, BPF_W);
>> + break;
>> + }
>> + /* PC += off if dst == src */
>> + /* PC += off if dst > src */
>> + /* PC += off if dst >= src */
>> + /* PC += off if dst != src */
>> + /* PC += off if dst > src (signed) */
>> + /* PC += off if dst >= src (signed) */
>> + /* PC += off if dst & src */
>> + case BPF_JMP | BPF_JEQ | BPF_X:
>> + case BPF_JMP | BPF_JGT | BPF_X:
>> + case BPF_JMP | BPF_JGE | BPF_X:
>> + case BPF_JMP | BPF_JNE | BPF_X:
>> + case BPF_JMP | BPF_JSGT | BPF_X:
>> + case BPF_JMP | BPF_JSGE | BPF_X:
>> + case BPF_JMP | BPF_JSET | BPF_X:
>> + /* Setup source registers */
>> + rm = sstk ? tmp2[0] : src_hi;
>> + rn = sstk ? tmp2[1] : src_lo;
>> + if (sstk) {
>> + emit(ARM_LDR_I(rn, ARM_SP, STACK_VAR(src_lo)), ctx);
>> + emit(ARM_LDR_I(rm, ARM_SP, STACK_VAR(src_hi)), ctx);
>> + }
>> + goto go_jmp;
>> + /* PC += off if dst == imm */
>> + /* PC += off if dst > imm */
>> + /* PC += off if dst >= imm */
>> + /* PC += off if dst != imm */
>> + /* PC += off if dst > imm (signed) */
>> + /* PC += off if dst >= imm (signed) */
>> + /* PC += off if dst & imm */
>> + case BPF_JMP | BPF_JEQ | BPF_K:
>> + case BPF_JMP | BPF_JGT | BPF_K:
>> + case BPF_JMP | BPF_JGE | BPF_K:
>> + case BPF_JMP | BPF_JNE | BPF_K:
>> + case BPF_JMP | BPF_JSGT | BPF_K:
>> + case BPF_JMP | BPF_JSGE | BPF_K:
>> + case BPF_JMP | BPF_JSET | BPF_K:
>> + if (off == 0)
>> break;
>> - case BPF_ANC | SKF_AD_PROTOCOL:
>> - /* A = ntohs(skb->protocol) */
>> - ctx->seen |= SEEN_SKB;
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff,
>> - protocol) != 2);
>> - off = offsetof(struct sk_buff, protocol);
>> - emit(ARM_LDRH_I(r_scratch, r_skb, off), ctx);
>> - emit_swap16(r_A, r_scratch, ctx);
>> + rm = tmp2[0];
>> + rn = tmp2[1];
>> + /* Sign-extend immediate value */
>> + emit_a32_mov_i64(true, tmp2, imm, false, ctx);
>> +go_jmp:
>> + /* Setup destination register */
>> + rd = dstk ? tmp[0] : dst_hi;
>> + rt = dstk ? tmp[1] : dst_lo;
>> + if (dstk) {
>> + emit(ARM_LDR_I(rt, ARM_SP, STACK_VAR(dst_lo)), ctx);
>> + emit(ARM_LDR_I(rd, ARM_SP, STACK_VAR(dst_hi)), ctx);
>> + }
>> +
>> + /* Check for the condition */
>> + emit_ar_r(rd, rt, rm, rn, ctx, BPF_OP(code));
>> +
>> + /* Setup JUMP instruction */
>> + jmp_offset = bpf2a32_offset(i+off, i, ctx);
>> + switch (BPF_OP(code)) {
>> + case BPF_JNE:
>> + case BPF_JSET:
>> + _emit(ARM_COND_NE, ARM_B(jmp_offset), ctx);
>> break;
>> - case BPF_ANC | SKF_AD_CPU:
>> - /* r_scratch = current_thread_info() */
>> - OP_IMM3(ARM_BIC, r_scratch, ARM_SP, THREAD_SIZE - 1, ctx);
>> - /* A = current_thread_info()->cpu */
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct thread_info, cpu) != 4);
>> - off = offsetof(struct thread_info, cpu);
>> - emit(ARM_LDR_I(r_A, r_scratch, off), ctx);
>> + case BPF_JEQ:
>> + _emit(ARM_COND_EQ, ARM_B(jmp_offset), ctx);
>> break;
>> - case BPF_ANC | SKF_AD_IFINDEX:
>> - case BPF_ANC | SKF_AD_HATYPE:
>> - /* A = skb->dev->ifindex */
>> - /* A = skb->dev->type */
>> - ctx->seen |= SEEN_SKB;
>> - off = offsetof(struct sk_buff, dev);
>> - emit(ARM_LDR_I(r_scratch, r_skb, off), ctx);
>> -
>> - emit(ARM_CMP_I(r_scratch, 0), ctx);
>> - emit_err_ret(ARM_COND_EQ, ctx);
>> -
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct net_device,
>> - ifindex) != 4);
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct net_device,
>> - type) != 2);
>> -
>> - if (code == (BPF_ANC | SKF_AD_IFINDEX)) {
>> - off = offsetof(struct net_device, ifindex);
>> - emit(ARM_LDR_I(r_A, r_scratch, off), ctx);
>> - } else {
>> - /*
>> - * offset of field "type" in "struct
>> - * net_device" is above what can be
>> - * used in the ldrh rd, [rn, #imm]
>> - * instruction, so load the offset in
>> - * a register and use ldrh rd, [rn, rm]
>> - */
>> - off = offsetof(struct net_device, type);
>> - emit_mov_i(ARM_R3, off, ctx);
>> - emit(ARM_LDRH_R(r_A, r_scratch, ARM_R3), ctx);
>> - }
>> + case BPF_JGT:
>> + _emit(ARM_COND_HI, ARM_B(jmp_offset), ctx);
>> break;
>> - case BPF_ANC | SKF_AD_MARK:
>> - ctx->seen |= SEEN_SKB;
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, mark) != 4);
>> - off = offsetof(struct sk_buff, mark);
>> - emit(ARM_LDR_I(r_A, r_skb, off), ctx);
>> + case BPF_JGE:
>> + _emit(ARM_COND_CS, ARM_B(jmp_offset), ctx);
>> break;
>> - case BPF_ANC | SKF_AD_RXHASH:
>> - ctx->seen |= SEEN_SKB;
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, hash) != 4);
>> - off = offsetof(struct sk_buff, hash);
>> - emit(ARM_LDR_I(r_A, r_skb, off), ctx);
>> + case BPF_JSGT:
>> + _emit(ARM_COND_LT, ARM_B(jmp_offset), ctx);
>> break;
>> - case BPF_ANC | SKF_AD_VLAN_TAG:
>> - case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
>> - ctx->seen |= SEEN_SKB;
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, vlan_tci) != 2);
>> - off = offsetof(struct sk_buff, vlan_tci);
>> - emit(ARM_LDRH_I(r_A, r_skb, off), ctx);
>> - if (code == (BPF_ANC | SKF_AD_VLAN_TAG))
>> - OP_IMM3(ARM_AND, r_A, r_A, ~VLAN_TAG_PRESENT, ctx);
>> - else {
>> - OP_IMM3(ARM_LSR, r_A, r_A, 12, ctx);
>> - OP_IMM3(ARM_AND, r_A, r_A, 0x1, ctx);
>> - }
>> + case BPF_JSGE:
>> + _emit(ARM_COND_GE, ARM_B(jmp_offset), ctx);
>> break;
>> - case BPF_ANC | SKF_AD_PKTTYPE:
>> - ctx->seen |= SEEN_SKB;
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff,
>> - __pkt_type_offset[0]) != 1);
>> - off = PKT_TYPE_OFFSET();
>> - emit(ARM_LDRB_I(r_A, r_skb, off), ctx);
>> - emit(ARM_AND_I(r_A, r_A, PKT_TYPE_MAX), ctx);
>> -#ifdef __BIG_ENDIAN_BITFIELD
>> - emit(ARM_LSR_I(r_A, r_A, 5), ctx);
>> -#endif
>> + }
>> + break;
>> + /* JMP OFF */
>> + case BPF_JMP | BPF_JA:
>> + {
>> + if (off == 0)
>> break;
>> - case BPF_ANC | SKF_AD_QUEUE:
>> - ctx->seen |= SEEN_SKB;
>> - BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff,
>> - queue_mapping) != 2);
>> - BUILD_BUG_ON(offsetof(struct sk_buff,
>> - queue_mapping) > 0xff);
>> - off = offsetof(struct sk_buff, queue_mapping);
>> - emit(ARM_LDRH_I(r_A, r_skb, off), ctx);
>> + jmp_offset = bpf2a32_offset(i+off, i, ctx);
>> + check_imm24(jmp_offset);
>> + emit(ARM_B(jmp_offset), ctx);
>> + break;
>> + }
>> + /* tail call */
>> + case BPF_JMP | BPF_CALL | BPF_X:
>> + if (emit_bpf_tail_call(ctx))
>> + return -EFAULT;
>> + break;
>> + /* function call */
>> + case BPF_JMP | BPF_CALL:
>> + goto notyet;
>> + /* function return */
>> + case BPF_JMP | BPF_EXIT:
>> + /* Optimization: when last instruction is EXIT
>> + * simply fallthrough to epilogue.
>> + */
>> + if (i == ctx->prog->len - 1)
>> break;
>> - case BPF_ANC | SKF_AD_PAY_OFFSET:
>> - ctx->seen |= SEEN_SKB | SEEN_CALL;
>> + jmp_offset = epilogue_offset(ctx);
>> + check_imm24(jmp_offset);
>> + emit(ARM_B(jmp_offset), ctx);
>> + break;
>> +notyet:
>> + pr_info_once("*** NOT YET: opcode %02x ***\n", code);
>> + return -EFAULT;
>> + default:
>> + pr_err_once("unknown opcode %02x\n", code);
>> + return -EINVAL;
>> + }
>>
>> - emit(ARM_MOV_R(ARM_R0, r_skb), ctx);
>> - emit_mov_i(ARM_R3, (unsigned int)skb_get_poff, ctx);
>> - emit_blx_r(ARM_R3, ctx);
>> - emit(ARM_MOV_R(r_A, ARM_R0), ctx);
>> - break;
>> - case BPF_LDX | BPF_W | BPF_ABS:
>> - /*
>> - * load a 32bit word from struct seccomp_data.
>> - * seccomp_check_filter() will already have checked
>> - * that k is 32bit aligned and lies within the
>> - * struct seccomp_data.
>> - */
>> - ctx->seen |= SEEN_SKB;
>> - emit(ARM_LDR_I(r_A, r_skb, k), ctx);
>> - break;
>> - default:
>> - return -1;
>> + if (ctx->flags & FLAG_IMM_OVERFLOW)
>> + /*
>> + * this instruction generated an overflow when
>> + * trying to access the literal pool, so
>> + * delegate this filter to the kernel interpreter.
>> + */
>> + return -1;
>> + return 0;
>> +}
>> +
>> +static int build_body(struct jit_ctx *ctx)
>> +{
>> + const struct bpf_prog *prog = ctx->prog;
>> + unsigned int i;
>> +
>> + for (i = 0; i < prog->len; i++) {
>> + const struct bpf_insn *insn = &(prog->insnsi[i]);
>> + int ret;
>> +
>> + ret = build_insn(insn, ctx);
>> +
>> + /* It's used with loading the 64 bit immediate value. */
>> + if (ret > 0) {
>> + i++;
>> + if (ctx->target == NULL)
>> + ctx->offsets[i] = ctx->idx;
>> + continue;
>> }
>>
>> - if (ctx->flags & FLAG_IMM_OVERFLOW)
>> - /*
>> - * this instruction generated an overflow when
>> - * trying to access the literal pool, so
>> - * delegate this filter to the kernel interpreter.
>> - */
>> - return -1;
>> + if (ctx->target == NULL)
>> + ctx->offsets[i] = ctx->idx;
>> +
>> + /* If unsuccesfull, return with error code */
>> + if (ret)
>> + return ret;
>> }
>> + return 0;
>> +}
>>
>> - /* compute offsets only during the first pass */
>> - if (ctx->target == NULL)
>> - ctx->offsets[i] = ctx->idx * 4;
>> +static int validate_code(struct jit_ctx *ctx)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < ctx->idx; i++) {
>> + u32 a32_insn = le32_to_cpu(ctx->target[i]);
>> +
>> + if (a32_insn == ARM_INST_UDF)
>> + return -1;
>> + }
>>
>> return 0;
>> }
>>
>> +void bpf_jit_compile(struct bpf_prog *prog)
>> +{
>> + /* Nothing to do here. We support Internal BPF. */
>> +}
>>
>> -void bpf_jit_compile(struct bpf_prog *fp)
>> +struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
>> {
>> +#ifdef __LITTLE_ENDIAN
>> + struct bpf_prog *tmp, *orig_prog = prog;
>> struct bpf_binary_header *header;
>> + bool tmp_blinded = false;
>> struct jit_ctx ctx;
>> - unsigned tmp_idx;
>> - unsigned alloc_size;
>> - u8 *target_ptr;
>> + unsigned int tmp_idx;
>> + unsigned int image_size;
>> + u8 *image_ptr;
>>
>> + /* If BPF JIT was not enabled then we must fall back to
>> + * the interpreter.
>> + */
>> if (!bpf_jit_enable)
>> - return;
>> + return orig_prog;
>>
>> - memset(&ctx, 0, sizeof(ctx));
>> - ctx.skf = fp;
>> - ctx.ret0_fp_idx = -1;
>> + /* If constant blinding was enabled and we failed during blinding
>> + * then we must fall back to the interpreter. Otherwise, we save
>> + * the new JITed code.
>> + */
>> + tmp = bpf_jit_blind_constants(prog);
>>
>> - ctx.offsets = kzalloc(4 * (ctx.skf->len + 1), GFP_KERNEL);
>> - if (ctx.offsets == NULL)
>> - return;
>> + if (IS_ERR(tmp))
>> + return orig_prog;
>> + if (tmp != prog) {
>> + tmp_blinded = true;
>> + prog = tmp;
>> + }
>> +
>> + memset(&ctx, 0, sizeof(ctx));
>> + ctx.prog = prog;
>>
>> - /* fake pass to fill in the ctx->seen */
>> - if (unlikely(build_body(&ctx)))
>> + /* Not able to allocate memory for offsets[] , then
>> + * we must fall back to the interpreter
>> + */
>> + ctx.offsets = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
>> + if (ctx.offsets == NULL) {
>> + prog = orig_prog;
>> goto out;
>> + }
>> +
>> + /* 1) fake pass to find in the length of the JITed code,
>> + * to compute ctx->offsets and other context variables
>> + * needed to compute final JITed code.
>> + * Also, calculate random starting pointer/start of JITed code
>> + * which is prefixed by random number of fault instructions.
>> + *
>> + * If the first pass fails then there is no chance of it
>> + * being successful in the second pass, so just fall back
>> + * to the interpreter.
>> + */
>> + if (build_body(&ctx)) {
>> + prog = orig_prog;
>> + goto out_off;
>> + }
>>
>> tmp_idx = ctx.idx;
>> build_prologue(&ctx);
>> ctx.prologue_bytes = (ctx.idx - tmp_idx) * 4;
>>
>> + ctx.epilogue_offset = ctx.idx;
>> +
>> #if __LINUX_ARM_ARCH__ < 7
>> tmp_idx = ctx.idx;
>> build_epilogue(&ctx);
>> @@ -1020,64 +1838,96 @@ void bpf_jit_compile(struct bpf_prog *fp)
>>
>> ctx.idx += ctx.imm_count;
>> if (ctx.imm_count) {
>> - ctx.imms = kzalloc(4 * ctx.imm_count, GFP_KERNEL);
>> - if (ctx.imms == NULL)
>> - goto out;
>> + ctx.imms = kcalloc(ctx.imm_count, sizeof(u32), GFP_KERNEL);
>> + if (ctx.imms == NULL) {
>> + prog = orig_prog;
>> + goto out_off;
>> + }
>> }
>> #else
>> - /* there's nothing after the epilogue on ARMv7 */
>> + /* there's nothing about the epilogue on ARMv7 */
>> build_epilogue(&ctx);
>> #endif
>> - alloc_size = 4 * ctx.idx;
>> - header = bpf_jit_binary_alloc(alloc_size, &target_ptr,
>> - 4, jit_fill_hole);
>> - if (header == NULL)
>> - goto out;
>> + /* Now we can get the actual image size of the JITed arm code.
>> + * Currently, we are not considering the THUMB-2 instructions
>> + * for jit, although it can decrease the size of the image.
>> + *
>> + * As each arm instruction is of length 32bit, we are translating
>> + * number of JITed intructions into the size required to store these
>> + * JITed code.
>> + */
>> + image_size = sizeof(u32) * ctx.idx;
>>
>> - ctx.target = (u32 *) target_ptr;
>> + /* Now we know the size of the structure to make */
>> + header = bpf_jit_binary_alloc(image_size, &image_ptr,
>> + sizeof(u32), jit_fill_hole);
>> + /* Not able to allocate memory for the structure then
>> + * we must fall back to the interpretation
>> + */
>> + if (header == NULL) {
>> + prog = orig_prog;
>> + goto out_imms;
>> + }
>> +
>> + /* 2.) Actual pass to generate final JIT code */
>> + ctx.target = (u32 *) image_ptr;
>> ctx.idx = 0;
>>
>> build_prologue(&ctx);
>> +
>> + /* If building the body of the JITed code fails somehow,
>> + * we fall back to the interpretation.
>> + */
>> if (build_body(&ctx) < 0) {
>> -#if __LINUX_ARM_ARCH__ < 7
>> - if (ctx.imm_count)
>> - kfree(ctx.imms);
>> -#endif
>> + image_ptr = NULL;
>> bpf_jit_binary_free(header);
>> - goto out;
>> + prog = orig_prog;
>> + goto out_imms;
>> }
>> build_epilogue(&ctx);
>>
>> + /* 3.) Extra pass to validate JITed Code */
>> + if (validate_code(&ctx)) {
>> + image_ptr = NULL;
>> + bpf_jit_binary_free(header);
>> + prog = orig_prog;
>> + goto out_imms;
>> + }
>> flush_icache_range((u32)header, (u32)(ctx.target + ctx.idx));
>>
>> -#if __LINUX_ARM_ARCH__ < 7
>> - if (ctx.imm_count)
>> - kfree(ctx.imms);
>> -#endif
>> -
>> if (bpf_jit_enable > 1)
>> /* there are 2 passes here */
>> - bpf_jit_dump(fp->len, alloc_size, 2, ctx.target);
>> + bpf_jit_dump(prog->len, image_size, 2, ctx.target);
>>
>> set_memory_ro((unsigned long)header, header->pages);
>> - fp->bpf_func = (void *)ctx.target;
>> - fp->jited = 1;
>> -out:
>> + prog->bpf_func = (void *)ctx.target;
>> + prog->jited = 1;
>> +out_imms:
>> +#if __LINUX_ARM_ARCH__ < 7
>> + if (ctx.imm_count)
>> + kfree(ctx.imms);
>> +#endif
>> +out_off:
>> kfree(ctx.offsets);
>> - return;
>> +out:
>> + if (tmp_blinded)
>> + bpf_jit_prog_release_other(prog, prog == orig_prog ?
>> + tmp : orig_prog);
>> +#endif /* __LITTLE_ENDIAN */
>> + return prog;
>> }
>>
>> -void bpf_jit_free(struct bpf_prog *fp)
>> +void bpf_jit_free(struct bpf_prog *prog)
>> {
>> - unsigned long addr = (unsigned long)fp->bpf_func & PAGE_MASK;
>> + unsigned long addr = (unsigned long)prog->bpf_func & PAGE_MASK;
>> struct bpf_binary_header *header = (void *)addr;
>>
>> - if (!fp->jited)
>> + if (!prog->jited)
>> goto free_filter;
>>
>> set_memory_rw(addr, header->pages);
>> bpf_jit_binary_free(header);
>>
>> free_filter:
>> - bpf_prog_unlock_free(fp);
>> + bpf_prog_unlock_free(prog);
>> }
>> diff --git a/arch/arm/net/bpf_jit_32.h b/arch/arm/net/bpf_jit_32.h
>> index c46fca2..d5cf5f6 100644
>> --- a/arch/arm/net/bpf_jit_32.h
>> +++ b/arch/arm/net/bpf_jit_32.h
>> @@ -11,6 +11,7 @@
>> #ifndef PFILTER_OPCODES_ARM_H
>> #define PFILTER_OPCODES_ARM_H
>>
>> +/* ARM 32bit Registers */
>> #define ARM_R0 0
>> #define ARM_R1 1
>> #define ARM_R2 2
>> @@ -22,38 +23,43 @@
>> #define ARM_R8 8
>> #define ARM_R9 9
>> #define ARM_R10 10
>> -#define ARM_FP 11
>> -#define ARM_IP 12
>> -#define ARM_SP 13
>> -#define ARM_LR 14
>> -#define ARM_PC 15
>> -
>> -#define ARM_COND_EQ 0x0
>> -#define ARM_COND_NE 0x1
>> -#define ARM_COND_CS 0x2
>> +#define ARM_FP 11 /* Frame Pointer */
>> +#define ARM_IP 12 /* Intra-procedure scratch register */
>> +#define ARM_SP 13 /* Stack pointer: as load/store base reg */
>> +#define ARM_LR 14 /* Link Register */
>> +#define ARM_PC 15 /* Program counter */
>> +
>> +#define ARM_COND_EQ 0x0 /* == */
>> +#define ARM_COND_NE 0x1 /* != */
>> +#define ARM_COND_CS 0x2 /* unsigned >= */
>> #define ARM_COND_HS ARM_COND_CS
>> -#define ARM_COND_CC 0x3
>> +#define ARM_COND_CC 0x3 /* unsigned < */
>> #define ARM_COND_LO ARM_COND_CC
>> -#define ARM_COND_MI 0x4
>> -#define ARM_COND_PL 0x5
>> -#define ARM_COND_VS 0x6
>> -#define ARM_COND_VC 0x7
>> -#define ARM_COND_HI 0x8
>> -#define ARM_COND_LS 0x9
>> -#define ARM_COND_GE 0xa
>> -#define ARM_COND_LT 0xb
>> -#define ARM_COND_GT 0xc
>> -#define ARM_COND_LE 0xd
>> -#define ARM_COND_AL 0xe
>> +#define ARM_COND_MI 0x4 /* < 0 */
>> +#define ARM_COND_PL 0x5 /* >= 0 */
>> +#define ARM_COND_VS 0x6 /* Signed Overflow */
>> +#define ARM_COND_VC 0x7 /* No Signed Overflow */
>> +#define ARM_COND_HI 0x8 /* unsigned > */
>> +#define ARM_COND_LS 0x9 /* unsigned <= */
>> +#define ARM_COND_GE 0xa /* Signed >= */
>> +#define ARM_COND_LT 0xb /* Signed < */
>> +#define ARM_COND_GT 0xc /* Signed > */
>> +#define ARM_COND_LE 0xd /* Signed <= */
>> +#define ARM_COND_AL 0xe /* None */
>>
>> /* register shift types */
>> #define SRTYPE_LSL 0
>> #define SRTYPE_LSR 1
>> #define SRTYPE_ASR 2
>> #define SRTYPE_ROR 3
>> +#define SRTYPE_ASL (SRTYPE_LSL)
>>
>> #define ARM_INST_ADD_R 0x00800000
>> +#define ARM_INST_ADDS_R 0x00900000
>> +#define ARM_INST_ADC_R 0x00a00000
>> +#define ARM_INST_ADC_I 0x02a00000
>> #define ARM_INST_ADD_I 0x02800000
>> +#define ARM_INST_ADDS_I 0x02900000
>>
>> #define ARM_INST_AND_R 0x00000000
>> #define ARM_INST_AND_I 0x02000000
>> @@ -76,8 +82,10 @@
>> #define ARM_INST_LDRH_I 0x01d000b0
>> #define ARM_INST_LDRH_R 0x019000b0
>> #define ARM_INST_LDR_I 0x05900000
>> +#define ARM_INST_LDR_R 0x07900000
>>
>> #define ARM_INST_LDM 0x08900000
>> +#define ARM_INST_LDM_IA 0x08b00000
>>
>> #define ARM_INST_LSL_I 0x01a00000
>> #define ARM_INST_LSL_R 0x01a00010
>> @@ -86,6 +94,7 @@
>> #define ARM_INST_LSR_R 0x01a00030
>>
>> #define ARM_INST_MOV_R 0x01a00000
>> +#define ARM_INST_MOVS_R 0x01b00000
>> #define ARM_INST_MOV_I 0x03a00000
>> #define ARM_INST_MOVW 0x03000000
>> #define ARM_INST_MOVT 0x03400000
>> @@ -96,17 +105,28 @@
>> #define ARM_INST_PUSH 0x092d0000
>>
>> #define ARM_INST_ORR_R 0x01800000
>> +#define ARM_INST_ORRS_R 0x01900000
>> #define ARM_INST_ORR_I 0x03800000
>>
>> #define ARM_INST_REV 0x06bf0f30
>> #define ARM_INST_REV16 0x06bf0fb0
>>
>> #define ARM_INST_RSB_I 0x02600000
>> +#define ARM_INST_RSBS_I 0x02700000
>> +#define ARM_INST_RSC_I 0x02e00000
>>
>> #define ARM_INST_SUB_R 0x00400000
>> +#define ARM_INST_SUBS_R 0x00500000
>> +#define ARM_INST_RSB_R 0x00600000
>> #define ARM_INST_SUB_I 0x02400000
>> +#define ARM_INST_SUBS_I 0x02500000
>> +#define ARM_INST_SBC_I 0x02c00000
>> +#define ARM_INST_SBC_R 0x00c00000
>> +#define ARM_INST_SBCS_R 0x00d00000
>>
>> #define ARM_INST_STR_I 0x05800000
>> +#define ARM_INST_STRB_I 0x05c00000
>> +#define ARM_INST_STRH_I 0x01c000b0
>>
>> #define ARM_INST_TST_R 0x01100000
>> #define ARM_INST_TST_I 0x03100000
>> @@ -117,6 +137,8 @@
>>
>> #define ARM_INST_MLS 0x00600090
>>
>> +#define ARM_INST_UXTH 0x06ff0070
>> +
>> /*
>> * Use a suitable undefined instruction to use for ARM/Thumb2 faulting.
>> * We need to be careful not to conflict with those used by other modules
>> @@ -135,9 +157,15 @@
>> #define _AL3_R(op, rd, rn, rm) ((op ## _R) | (rd) << 12 | (rn) << 16 | (rm))
>> /* immediate */
>> #define _AL3_I(op, rd, rn, imm) ((op ## _I) | (rd) << 12 | (rn) << 16 | (imm))
>> +/* register with register-shift */
>> +#define _AL3_SR(inst) (inst | (1 << 4))
>>
>> #define ARM_ADD_R(rd, rn, rm) _AL3_R(ARM_INST_ADD, rd, rn, rm)
>> +#define ARM_ADDS_R(rd, rn, rm) _AL3_R(ARM_INST_ADDS, rd, rn, rm)
>> #define ARM_ADD_I(rd, rn, imm) _AL3_I(ARM_INST_ADD, rd, rn, imm)
>> +#define ARM_ADDS_I(rd, rn, imm) _AL3_I(ARM_INST_ADDS, rd, rn, imm)
>> +#define ARM_ADC_R(rd, rn, rm) _AL3_R(ARM_INST_ADC, rd, rn, rm)
>> +#define ARM_ADC_I(rd, rn, imm) _AL3_I(ARM_INST_ADC, rd, rn, imm)
>>
>> #define ARM_AND_R(rd, rn, rm) _AL3_R(ARM_INST_AND, rd, rn, rm)
>> #define ARM_AND_I(rd, rn, imm) _AL3_I(ARM_INST_AND, rd, rn, imm)
>> @@ -156,7 +184,9 @@
>> #define ARM_EOR_I(rd, rn, imm) _AL3_I(ARM_INST_EOR, rd, rn, imm)
>>
>> #define ARM_LDR_I(rt, rn, off) (ARM_INST_LDR_I | (rt) << 12 | (rn) << 16 \
>> - | (off))
>> + | ((off) & 0xfff))
>> +#define ARM_LDR_R(rt, rn, rm) (ARM_INST_LDR_R | (rt) << 12 | (rn) << 16 \
>> + | (rm))
>> #define ARM_LDRB_I(rt, rn, off) (ARM_INST_LDRB_I | (rt) << 12 | (rn) << 16 \
>> | (off))
>> #define ARM_LDRB_R(rt, rn, rm) (ARM_INST_LDRB_R | (rt) << 12 | (rn) << 16 \
>> @@ -167,15 +197,23 @@
>> | (rm))
>>
>> #define ARM_LDM(rn, regs) (ARM_INST_LDM | (rn) << 16 | (regs))
>> +#define ARM_LDM_IA(rn, regs) (ARM_INST_LDM_IA | (rn) << 16 | (regs))
>>
>> #define ARM_LSL_R(rd, rn, rm) (_AL3_R(ARM_INST_LSL, rd, 0, rn) | (rm) << 8)
>> #define ARM_LSL_I(rd, rn, imm) (_AL3_I(ARM_INST_LSL, rd, 0, rn) | (imm) << 7)
>>
>> #define ARM_LSR_R(rd, rn, rm) (_AL3_R(ARM_INST_LSR, rd, 0, rn) | (rm) << 8)
>> #define ARM_LSR_I(rd, rn, imm) (_AL3_I(ARM_INST_LSR, rd, 0, rn) | (imm) << 7)
>> +#define ARM_ASR_R(rd, rn, rm) (_AL3_R(ARM_INST_ASR, rd, 0, rn) | (rm) << 8)
>> +#define ARM_ASR_I(rd, rn, imm) (_AL3_I(ARM_INST_ASR, rd, 0, rn) | (imm) << 7)
>>
>> #define ARM_MOV_R(rd, rm) _AL3_R(ARM_INST_MOV, rd, 0, rm)
>> +#define ARM_MOVS_R(rd, rm) _AL3_R(ARM_INST_MOVS, rd, 0, rm)
>> #define ARM_MOV_I(rd, imm) _AL3_I(ARM_INST_MOV, rd, 0, imm)
>> +#define ARM_MOV_SR(rd, rm, type, rs) \
>> + (_AL3_SR(ARM_MOV_R(rd, rm)) | (type) << 5 | (rs) << 8)
>> +#define ARM_MOV_SI(rd, rm, type, imm6) \
>> + (ARM_MOV_R(rd, rm) | (type) << 5 | (imm6) << 7)
>>
>> #define ARM_MOVW(rd, imm) \
>> (ARM_INST_MOVW | ((imm) >> 12) << 16 | (rd) << 12 | ((imm) & 0x0fff))
>> @@ -190,19 +228,38 @@
>>
>> #define ARM_ORR_R(rd, rn, rm) _AL3_R(ARM_INST_ORR, rd, rn, rm)
>> #define ARM_ORR_I(rd, rn, imm) _AL3_I(ARM_INST_ORR, rd, rn, imm)
>> -#define ARM_ORR_S(rd, rn, rm, type, rs) \
>> - (ARM_ORR_R(rd, rn, rm) | (type) << 5 | (rs) << 7)
>> +#define ARM_ORR_SR(rd, rn, rm, type, rs) \
>> + (_AL3_SR(ARM_ORR_R(rd, rn, rm)) | (type) << 5 | (rs) << 8)
>> +#define ARM_ORRS_R(rd, rn, rm) _AL3_R(ARM_INST_ORRS, rd, rn, rm)
>> +#define ARM_ORRS_SR(rd, rn, rm, type, rs) \
>> + (_AL3_SR(ARM_ORRS_R(rd, rn, rm)) | (type) << 5 | (rs) << 8)
>> +#define ARM_ORR_SI(rd, rn, rm, type, imm6) \
>> + (ARM_ORR_R(rd, rn, rm) | (type) << 5 | (imm6) << 7)
>> +#define ARM_ORRS_SI(rd, rn, rm, type, imm6) \
>> + (ARM_ORRS_R(rd, rn, rm) | (type) << 5 | (imm6) << 7)
>>
>> #define ARM_REV(rd, rm) (ARM_INST_REV | (rd) << 12 | (rm))
>> #define ARM_REV16(rd, rm) (ARM_INST_REV16 | (rd) << 12 | (rm))
>>
>> #define ARM_RSB_I(rd, rn, imm) _AL3_I(ARM_INST_RSB, rd, rn, imm)
>> +#define ARM_RSBS_I(rd, rn, imm) _AL3_I(ARM_INST_RSBS, rd, rn, imm)
>> +#define ARM_RSC_I(rd, rn, imm) _AL3_I(ARM_INST_RSC, rd, rn, imm)
>>
>> #define ARM_SUB_R(rd, rn, rm) _AL3_R(ARM_INST_SUB, rd, rn, rm)
>> +#define ARM_SUBS_R(rd, rn, rm) _AL3_R(ARM_INST_SUBS, rd, rn, rm)
>> +#define ARM_RSB_R(rd, rn, rm) _AL3_R(ARM_INST_RSB, rd, rn, rm)
>> +#define ARM_SBC_R(rd, rn, rm) _AL3_R(ARM_INST_SBC, rd, rn, rm)
>> +#define ARM_SBCS_R(rd, rn, rm) _AL3_R(ARM_INST_SBCS, rd, rn, rm)
>> #define ARM_SUB_I(rd, rn, imm) _AL3_I(ARM_INST_SUB, rd, rn, imm)
>> +#define ARM_SUBS_I(rd, rn, imm) _AL3_I(ARM_INST_SUBS, rd, rn, imm)
>> +#define ARM_SBC_I(rd, rn, imm) _AL3_I(ARM_INST_SBC, rd, rn, imm)
>>
>> #define ARM_STR_I(rt, rn, off) (ARM_INST_STR_I | (rt) << 12 | (rn) << 16 \
>> - | (off))
>> + | ((off) & 0xfff))
>> +#define ARM_STRH_I(rt, rn, off) (ARM_INST_STRH_I | (rt) << 12 | (rn) << 16 \
>> + | (((off) & 0xf0) << 4) | ((off) & 0xf))
>> +#define ARM_STRB_I(rt, rn, off) (ARM_INST_STRB_I | (rt) << 12 | (rn) << 16 \
>> + | (((off) & 0xf0) << 4) | ((off) & 0xf))
>>
>> #define ARM_TST_R(rn, rm) _AL3_R(ARM_INST_TST, 0, rn, rm)
>> #define ARM_TST_I(rn, imm) _AL3_I(ARM_INST_TST, 0, rn, imm)
>> @@ -214,5 +271,6 @@
>>
>> #define ARM_MLS(rd, rn, rm, ra) (ARM_INST_MLS | (rd) << 16 | (rn) | (rm) << 8 \
>> | (ra) << 12)
>> +#define ARM_UXTH(rd, rm) (ARM_INST_UXTH | (rd) << 12 | (rm))
>>
>> #endif /* PFILTER_OPCODES_ARM_H */
>> --
>> 2.7.4
>>
>
>
>
> --
> Kees Cook
> Pixel Security
^ permalink raw reply
* Re: [PATCH net-next 00/16] nfp: ctrl vNIC
From: Jakub Kicinski @ 2017-06-06 19:54 UTC (permalink / raw)
To: Jiri Pirko; +Cc: netdev, oss-drivers
In-Reply-To: <20170606091757.GC1911@nanopsycho>
On Tue, 6 Jun 2017 11:17:57 +0200, Jiri Pirko wrote:
> >> >What were your plans with pre-netdev config?
> >>
> >> We need to pass come initial resource division. Generally the consensus
> >> is to have these options exposed through devlink, let the user configure
> >> them all and then to have a trigger that would cause driver
> >> re-orchestration according to the new values. The flow would look like
> >> this:
> >>
> >> -driver loads with defaults, inits hw and instantiates netdevs
> >> -driver exposes config options via devlink
> >> -user sets up the options
> >> -user pushes the "go" trigger
> >> -upon the trigger command, devlink calls the driver re-init callback
> >> -driver shuts down the current instances, re-initializes hw,
> >> re-instantiates the netdevs
> >>
> >> Makes sense?
> >
> >I like the idea of a "go"/apply/reload trigger and extending devlink.
> >Do you plan on adding a way to persist the settings? I'm concerned NIC
> >users may want to boot into the right mode once it's set, without
> >reloads and reconfigs upon boot. Also is there going to be a way to
> >query the pending/running config? Sounds like we may want to expose
> >three value sets - persistent/default, running and pending/to be
> >applied.
>
> I don't think it is a good idea to introduce any kind of configuration
> persistency in HW. I believe that user is the master and he has all
> needed info. He can store it persistently, but it is up to him.
>
> So basicaly during boot, we need the devlink configuration to happen
> early on, before the netdevices get configured. udev? Not sure how
> exactly to do this. Have to ask around :)
Happy to hear that. Now there is two of us, I'll try again with the
marketing dept :)
^ permalink raw reply
* Re: [PATCH net 3/3] netvsc: fix RCU warning from set_multicast
From: David Miller @ 2017-06-06 19:55 UTC (permalink / raw)
To: stephen; +Cc: netdev, sthemmin
In-Reply-To: <20170605211010.9571-4-sthemmin@microsoft.com>
From: Stephen Hemminger <stephen@networkplumber.org>
Date: Mon, 5 Jun 2017 14:10:10 -0700
> + nvdev = rtnl_dereference(ndevctx->nvdev);
> + if (nvdev)
> + rdev = nvdev->extension;
> +
> + if (rdev) {
> + if (ndev->flags & IFF_PROMISC)
> + rndis_filter_set_packet_filter(rdev,
> + NDIS_PACKET_TYPE_PROMISCUOUS);
> + else
> + rndis_filter_set_packet_filter(rdev,
> + NDIS_PACKET_TYPE_BROADCAST |
> + NDIS_PACKET_TYPE_ALL_MULTICAST |
> + NDIS_PACKET_TYPE_DIRECTED);
> + }
> + }
Stephen, please at least compile test your code.
This is getting rediculous.
^ permalink raw reply
* Re: [PATCH] ravb: Fix use-after-free on `ifconfig eth0 down`
From: David Miller @ 2017-06-06 20:03 UTC (permalink / raw)
To: erosca
Cc: sergei.shtylyov, horms+renesas, kazuya.mizuguchi.ks, netdev,
linux-renesas-soc
In-Reply-To: <20170605220810.3933-1-erosca@de.adit-jv.com>
From: Eugeniu Rosca <erosca@de.adit-jv.com>
Date: Tue, 6 Jun 2017 00:08:10 +0200
> Commit a47b70ea86bd ("ravb: unmap descriptors when freeing rings") has
> introduced the issue seen in [1] reproduced on H3ULCB board.
>
> Fix this by relocating the RX skb ringbuffer free operation, so that
> swiotlb page unmapping can be done first. Freeing of aligned TX buffers
> is not relevant to the issue seen in [1]. Still, reposition TX free
> calls as well, to have all kfree() operations performed consistently
> _after_ dma_unmap_*()/dma_free_*().
>
> [1] Console screenshot with the problem reproduced:
>
> salvator-x login: root
> root@salvator-x:~# ifconfig eth0 up
> Micrel KSZ9031 Gigabit PHY e6800000.ethernet-ffffffff:00: \
> attached PHY driver [Micrel KSZ9031 Gigabit PHY] \
> (mii_bus:phy_addr=e6800000.ethernet-ffffffff:00, irq=235)
> IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> root@salvator-x:~#
> root@salvator-x:~# ifconfig eth0 down
> ==================================================================
> BUG: KASAN: use-after-free in swiotlb_tbl_unmap_single+0xc4/0x35c
...
> ==================================================================
> Disabling lock debugging due to kernel taint
> root@salvator-x:~#
>
> Fixes: a47b70ea86bd ("ravb: unmap descriptors when freeing rings")
> Signed-off-by: Eugeniu Rosca <erosca@de.adit-jv.com>
Applied and queued up for -stable, thanks.
^ permalink raw reply
* Re: [PATCH net-next] net: dsa: mv88e6xxx: fix 6085 frame mode masking
From: David Miller @ 2017-06-06 20:04 UTC (permalink / raw)
To: vivien.didelot; +Cc: netdev, linux-kernel, kernel, f.fainelli, andrew
In-Reply-To: <20170605221716.32235-1-vivien.didelot@savoirfairelinux.com>
From: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
Date: Mon, 5 Jun 2017 18:17:16 -0400
> The register bits used for the frame mode were masked with DSA (0x1)
> instead of the mask value (0x3) in the 6085 implementation of
> port_set_frame_mode. Fix this.
>
> Fixes: 56995cbc3540 ("net: dsa: mv88e6xxx: Refactor CPU and DSA port setup")
> Signed-off-by: Vivien Didelot <vivien.didelot@savoirfairelinux.com>
Applied, thanks Vivien.
^ permalink raw reply
* Re: [PATCH net] net: bridge: fix a null pointer dereference in br_afspec
From: David Miller @ 2017-06-06 20:06 UTC (permalink / raw)
To: nikolay; +Cc: netdev, roopa, stephen, garsilva, bridge
In-Reply-To: <1496701584-26855-1-git-send-email-nikolay@cumulusnetworks.com>
From: Nikolay Aleksandrov <nikolay@cumulusnetworks.com>
Date: Tue, 6 Jun 2017 01:26:24 +0300
> We might call br_afspec() with p == NULL which is a valid use case if
> the action is on the bridge device itself, but the bridge tunnel code
> dereferences the p pointer without checking, so check if p is null
> first.
>
> Reported-by: Gustavo A. R. Silva <garsilva@embeddedor.com>
> Fixes: efa5356b0d97 ("bridge: per vlan dst_metadata netlink support")
> Signed-off-by: Nikolay Aleksandrov <nikolay@cumulusnetworks.com>
Applied and queued up for -stable, thanks Nikolay.
^ permalink raw reply
* Re: [PATCH 1/2] hsr: fix coding style issues
From: David Miller @ 2017-06-06 20:09 UTC (permalink / raw)
To: m-karicheri2; +Cc: arvid.brodin, netdev, linux-kernel
In-Reply-To: <1496701907-29376-1-git-send-email-m-karicheri2@ti.com>
Please do not mix cleanups with legitimate bug fixes. Also, when posting
a multi-patch series, you must always provide an appropriate "[PATCH 0/N]"
header posting that describes what you series is doing at a high level,
how it is doing it, and why it is doing it that way.
For this, submit the erroneous warning removal against 'net' as a single
patch. And then once that propagates into the 'net-next' tree you can
submit the coding style cleanups against 'net-next', thanks.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox