* NFS over NAT causes e1000e transmit hangs
From: Florian Fainelli @ 2017-04-18 18:18 UTC (permalink / raw)
To: intel-wired-lan, jeffrey.t.kirsher; +Cc: netdev
Hi,
I am using NFS over a NAT with two e1000e adapters and with eth1 being
the LAN interface and eth0 the WAN interface. The kernel is Ubuntu's
16.10 kernel: 4.8.0-46-generic. The device doing NAT over NFS is just
mounting a remote folder and doing normal execution/file accesses. It's
enough to untar a file from this device onto a NFS share to expose the
problem.
The transmit hangs look like the ones below, doing a rmmod/insmod does
not help eliminated the problem, nor does a power cycle. Stopping the
NFS over NAT definitively does let the adapter recover.
Happy to test any patches/newer kernels if you think there is something
obviously wrong. It *seems* to have started when I updated to 4.8.x, and
I was not able to see this under 4.4, so first things could be to try a
bisection, time permitting.
The two devices involved in the NAT are:
fainelli@fainelli-desktop:[~/../linux]$ lspci -s 0000:09:00.0 -v
09:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
Subsystem: Intel Corporation Gigabit CT Desktop Adapter
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at ef6c0000 (32-bit, non-prefetchable) [size=128K]
Memory at ef600000 (32-bit, non-prefetchable) [size=512K]
I/O ports at b000 [size=32]
Memory at ef6e0000 (32-bit, non-prefetchable) [size=16K]
Expansion ROM at ef680000 [disabled] [size=256K]
Capabilities: <access denied>
Kernel driver in use: e1000e
Kernel modules: e1000e
fainelli@fainelli-desktop:[~/../linux]$ lspci -s 0000:00:19.0 -v
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network
Connection (rev 05)
Subsystem: Dell 82579LM Gigabit Network Connection
Flags: bus master, fast devsel, latency 0, IRQ 43
Memory at ef900000 (32-bit, non-prefetchable) [size=128K]
Memory at ef929000 (32-bit, non-prefetchable) [size=4K]
I/O ports at f040 [size=32]
Capabilities: <access denied>
Kernel driver in use: e1000e
Kernel modules: e1000e
[516481.589090] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
TDH <9b>
TDT <b0>
next_to_use <b0>
next_to_clean <96>
buffer_info[next_to_clean]:
time_stamp <107b0fc76>
next_to_watch <9b>
jiffies <107b10048>
next_to_watch.status <0>
MAC Status <40080083>
PHY Status <796d>
PHY 1000BASE-T Status <3c00>
PHY Extended Status <3000>
PCI Status <10>
[516483.573120] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
TDH <9b>
TDT <b0>
next_to_use <b0>
next_to_clean <96>
buffer_info[next_to_clean]:
time_stamp <107b0fc76>
next_to_watch <9b>
jiffies <107b10238>
next_to_watch.status <0>
MAC Status <40080083>
PHY Status <796d>
PHY 1000BASE-T Status <3c00>
PHY Extended Status <3000>
PCI Status <10>
[516485.589452] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
TDH <9b>
TDT <b0>
next_to_use <b0>
next_to_clean <96>
buffer_info[next_to_clean]:
time_stamp <107b0fc76>
next_to_watch <9b>
jiffies <107b10430>
next_to_watch.status <0>
MAC Status <40080083>
PHY Status <796d>
PHY 1000BASE-T Status <3c00>
PHY Extended Status <3000>
PCI Status <10>
[516487.573397] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
TDH <9b>
TDT <b0>
next_to_use <b0>
next_to_clean <96>
buffer_info[next_to_clean]:
time_stamp <107b0fc76>
next_to_watch <9b>
jiffies <107b10620>
next_to_watch.status <0>
MAC Status <40080083>
PHY Status <796d>
PHY 1000BASE-T Status <3c00>
PHY Extended Status <3000>
PCI Status <10>
[516487.700509] e1000e 0000:00:19.0 eth0: Reset adapter unexpectedly
[516491.526799] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: Rx/Tx
Thanks for reading, here is a virtual potato: 0.
--
Florian
^ permalink raw reply
* Re: export pcie_flr and remove copies of it in drivers V2
From: Bjorn Helgaas @ 2017-04-18 18:36 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Bjorn Helgaas, Giovanni Cabiddu, Salvatore Benedetto,
Mike Marciniszyn, Dennis Dalessandro, Derek Chickles,
Satanand Burla, Felix Manlunas, Raghu Vatsavayi, Jeff Kirsher,
linux-pci, qat-linux, linux-crypto, linux-rdma, netdev,
linux-kernel
In-Reply-To: <20170414191131.14286-1-hch@lst.de>
On Fri, Apr 14, 2017 at 09:11:24PM +0200, Christoph Hellwig wrote:
> Hi all,
>
> this exports the PCI layer pcie_flr helper, and removes various opencoded
> copies of it.
>
> Changes since V1:
> - rebase on top of the pci/virtualization branch
> - fixed the probe case in __pci_dev_reset
> - added ACKs from Bjorn
Applied the first three patches:
bc13871ef35a PCI: Export pcie_flr()
e641c375d414 PCI: Call pcie_flr() from reset_intel_82599_sfp_virtfn()
40e0901ea4bf PCI: Call pcie_flr() from reset_chelsio_generic_dev()
to pci/virtualization for v4.12, thanks!
^ permalink raw reply
* Re: [PATCH RFC] sparc64: eBPF JIT
From: David Miller @ 2017-04-18 18:37 UTC (permalink / raw)
To: alexei.starovoitov; +Cc: sparclinux, netdev, ast, daniel
In-Reply-To: <20170418054444.GA36377@ast-mbp.thefacebook.com>
From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Date: Mon, 17 Apr 2017 22:44:47 -0700
> The way llvm generates stack access is:
> rX = r10
> rX += imm
> and that's the only thing verifier recognizes as valid ptr_to_stack.
> Like rX -= imm will not be recognized as proper stack offset,
> since llvm never does it.
That simplifies things significantly for me.
I only allow moves from the frame pointer to another register,
and when I see that I rewrite it to "add FP, STACK_BIAS, DST_REG"
^ permalink raw reply
* Re: [PATCH v3 net-next RFC] Generic XDP
From: David Miller @ 2017-04-18 18:46 UTC (permalink / raw)
To: alexei.starovoitov; +Cc: brouer, kubakici, netdev, xdp-newbies
In-Reply-To: <20170417230436.GA96258@ast-mbp.thefacebook.com>
From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Date: Mon, 17 Apr 2017 16:04:38 -0700
> On Mon, Apr 17, 2017 at 03:49:55PM -0400, David Miller wrote:
>> From: Jesper Dangaard Brouer <brouer@redhat.com>
>> Date: Sun, 16 Apr 2017 22:26:01 +0200
>>
>> > The bpf tail-call use-case is a very good example of why the
>> > verifier cannot deduct the needed HEADROOM upfront.
>>
>> This brings up a very interesting question for me.
>>
>> I notice that tail calls are implemented by JITs largely by skipping
>> over the prologue of that destination program.
>>
>> However, many JITs preload cached SKB values into fixed registers in
>> the prologue. But they only do this if the program being JITed needs
>> those values.
>>
>> So how can it work properly if a program that does not need the SKB
>> values tail calls into one that does?
>
> For x86 JIT it's fine, since caching of skb values is not part of the prologue:
> emit_prologue(&prog);
> if (seen_ld_abs)
> emit_load_skb_data_hlen(&prog);
> and tail_call jumps into the next program as:
> EMIT4(0x48, 0x83, 0xC0, PROLOGUE_SIZE); /* add rax, prologue_size */
> EMIT2(0xFF, 0xE0); /* jmp rax */
> whereas inside emit_prologue() we have:
> B UILD_BUG_ON(cnt != PROLOGUE_SIZE);
>
> arm64 has similar proplogue skipping code and it's even
> simpler than x86, since it doesn't try to optimize LD_ABS/IND in assembler
> and instead calls into bpf_load_pointer() from generated code,
> so no caching of skb values at all.
>
> s390 jit has partial skipping of prologue, since bunch
> of registers are save/restored during tail_call and it looks fine
> to me as well.
Ok, what about stack usage?
Currently if I don't see a reference to FP then I elide allocating
MAX_BPF_STACK stack space.
What if, with tail calls, some programs need that stack space whilst
other's done?
It looks like, for example, JITs like powerpc avoids this issue
because they allocate the full MAX_BPF_STACK all the time. That seems
like overkill to me and bad for cache locality.
^ permalink raw reply
* [PATCH net-next 0/2] sparc64 eBPF JIT
From: David Miller @ 2017-04-18 18:57 UTC (permalink / raw)
To: sparclinux; +Cc: netdev, ast, daniel
This series adds an eBPF JIT for sparc64.
Sparc32 keeps having it's cBPF JIT after these changes.
Thanks to Daniel and Alexei for their invaluable feedback and review.
Signed-off-by: David S. Miller <davem@davemloft.net>
---
since RFC:
-- rework register allocation so that we don't have to emit move
instructions to get the arguments into place for calls
-- allow FP as source for 64-bit ALU move operations
-- don't emit NOP instructions in prologue, instead do multiple
passes so that all the branch offsets settle down
-- remove unnecessary checks on imm64 loads that verifier does
-- consistently use bpf2sparc[] instead of direct register references
^ permalink raw reply
* [PATCH net-next 1/2] sparc: Split BPF JIT into 32-bit and 64-bit.
From: David Miller @ 2017-04-18 18:58 UTC (permalink / raw)
To: sparclinux; +Cc: netdev, ast, daniel
This is in preparation for adding the 64-bit eBPF JIT.
Signed-off-by: David S. Miller <davem@davemloft.net>
---
arch/sparc/net/Makefile | 2 +-
arch/sparc/net/bpf_jit.h | 68 ----
arch/sparc/net/bpf_jit_32.h | 68 ++++
arch/sparc/net/bpf_jit_asm.S | 208 ----------
arch/sparc/net/bpf_jit_asm_32.S | 208 ++++++++++
arch/sparc/net/bpf_jit_asm_64.S | 1 +
arch/sparc/net/bpf_jit_comp.c | 815 ---------------------------------------
arch/sparc/net/bpf_jit_comp_32.c | 815 +++++++++++++++++++++++++++++++++++++++
arch/sparc/net/bpf_jit_comp_64.c | 1 +
9 files changed, 1094 insertions(+), 1092 deletions(-)
delete mode 100644 arch/sparc/net/bpf_jit.h
create mode 100644 arch/sparc/net/bpf_jit_32.h
delete mode 100644 arch/sparc/net/bpf_jit_asm.S
create mode 100644 arch/sparc/net/bpf_jit_asm_32.S
create mode 100644 arch/sparc/net/bpf_jit_asm_64.S
delete mode 100644 arch/sparc/net/bpf_jit_comp.c
create mode 100644 arch/sparc/net/bpf_jit_comp_32.c
create mode 100644 arch/sparc/net/bpf_jit_comp_64.c
diff --git a/arch/sparc/net/Makefile b/arch/sparc/net/Makefile
index 1306a58..76fa8e9 100644
--- a/arch/sparc/net/Makefile
+++ b/arch/sparc/net/Makefile
@@ -1,4 +1,4 @@
#
# Arch-specific network modules
#
-obj-$(CONFIG_BPF_JIT) += bpf_jit_asm.o bpf_jit_comp.o
+obj-$(CONFIG_BPF_JIT) += bpf_jit_asm_$(BITS).o bpf_jit_comp_$(BITS).o
diff --git a/arch/sparc/net/bpf_jit.h b/arch/sparc/net/bpf_jit.h
deleted file mode 100644
index 33d6b37..0000000
--- a/arch/sparc/net/bpf_jit.h
+++ /dev/null
@@ -1,68 +0,0 @@
-#ifndef _BPF_JIT_H
-#define _BPF_JIT_H
-
-/* Conventions:
- * %g1 : temporary
- * %g2 : Secondary temporary used by SKB data helper stubs.
- * %g3 : packet offset passed into SKB data helper stubs.
- * %o0 : pointer to skb (first argument given to JIT function)
- * %o1 : BPF A accumulator
- * %o2 : BPF X accumulator
- * %o3 : Holds saved %o7 so we can call helper functions without needing
- * to allocate a register window.
- * %o4 : skb->len - skb->data_len
- * %o5 : skb->data
- */
-
-#ifndef __ASSEMBLER__
-#define G0 0x00
-#define G1 0x01
-#define G3 0x03
-#define G6 0x06
-#define O0 0x08
-#define O1 0x09
-#define O2 0x0a
-#define O3 0x0b
-#define O4 0x0c
-#define O5 0x0d
-#define SP 0x0e
-#define O7 0x0f
-#define FP 0x1e
-
-#define r_SKB O0
-#define r_A O1
-#define r_X O2
-#define r_saved_O7 O3
-#define r_HEADLEN O4
-#define r_SKB_DATA O5
-#define r_TMP G1
-#define r_TMP2 G2
-#define r_OFF G3
-
-/* assembly code in arch/sparc/net/bpf_jit_asm.S */
-extern u32 bpf_jit_load_word[];
-extern u32 bpf_jit_load_half[];
-extern u32 bpf_jit_load_byte[];
-extern u32 bpf_jit_load_byte_msh[];
-extern u32 bpf_jit_load_word_positive_offset[];
-extern u32 bpf_jit_load_half_positive_offset[];
-extern u32 bpf_jit_load_byte_positive_offset[];
-extern u32 bpf_jit_load_byte_msh_positive_offset[];
-extern u32 bpf_jit_load_word_negative_offset[];
-extern u32 bpf_jit_load_half_negative_offset[];
-extern u32 bpf_jit_load_byte_negative_offset[];
-extern u32 bpf_jit_load_byte_msh_negative_offset[];
-
-#else
-#define r_SKB %o0
-#define r_A %o1
-#define r_X %o2
-#define r_saved_O7 %o3
-#define r_HEADLEN %o4
-#define r_SKB_DATA %o5
-#define r_TMP %g1
-#define r_TMP2 %g2
-#define r_OFF %g3
-#endif
-
-#endif /* _BPF_JIT_H */
diff --git a/arch/sparc/net/bpf_jit_32.h b/arch/sparc/net/bpf_jit_32.h
new file mode 100644
index 0000000..33d6b37
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_32.h
@@ -0,0 +1,68 @@
+#ifndef _BPF_JIT_H
+#define _BPF_JIT_H
+
+/* Conventions:
+ * %g1 : temporary
+ * %g2 : Secondary temporary used by SKB data helper stubs.
+ * %g3 : packet offset passed into SKB data helper stubs.
+ * %o0 : pointer to skb (first argument given to JIT function)
+ * %o1 : BPF A accumulator
+ * %o2 : BPF X accumulator
+ * %o3 : Holds saved %o7 so we can call helper functions without needing
+ * to allocate a register window.
+ * %o4 : skb->len - skb->data_len
+ * %o5 : skb->data
+ */
+
+#ifndef __ASSEMBLER__
+#define G0 0x00
+#define G1 0x01
+#define G3 0x03
+#define G6 0x06
+#define O0 0x08
+#define O1 0x09
+#define O2 0x0a
+#define O3 0x0b
+#define O4 0x0c
+#define O5 0x0d
+#define SP 0x0e
+#define O7 0x0f
+#define FP 0x1e
+
+#define r_SKB O0
+#define r_A O1
+#define r_X O2
+#define r_saved_O7 O3
+#define r_HEADLEN O4
+#define r_SKB_DATA O5
+#define r_TMP G1
+#define r_TMP2 G2
+#define r_OFF G3
+
+/* assembly code in arch/sparc/net/bpf_jit_asm.S */
+extern u32 bpf_jit_load_word[];
+extern u32 bpf_jit_load_half[];
+extern u32 bpf_jit_load_byte[];
+extern u32 bpf_jit_load_byte_msh[];
+extern u32 bpf_jit_load_word_positive_offset[];
+extern u32 bpf_jit_load_half_positive_offset[];
+extern u32 bpf_jit_load_byte_positive_offset[];
+extern u32 bpf_jit_load_byte_msh_positive_offset[];
+extern u32 bpf_jit_load_word_negative_offset[];
+extern u32 bpf_jit_load_half_negative_offset[];
+extern u32 bpf_jit_load_byte_negative_offset[];
+extern u32 bpf_jit_load_byte_msh_negative_offset[];
+
+#else
+#define r_SKB %o0
+#define r_A %o1
+#define r_X %o2
+#define r_saved_O7 %o3
+#define r_HEADLEN %o4
+#define r_SKB_DATA %o5
+#define r_TMP %g1
+#define r_TMP2 %g2
+#define r_OFF %g3
+#endif
+
+#endif /* _BPF_JIT_H */
diff --git a/arch/sparc/net/bpf_jit_asm.S b/arch/sparc/net/bpf_jit_asm.S
deleted file mode 100644
index 8c83f4b..0000000
--- a/arch/sparc/net/bpf_jit_asm.S
+++ /dev/null
@@ -1,208 +0,0 @@
-#include <asm/ptrace.h>
-
-#include "bpf_jit.h"
-
-#ifdef CONFIG_SPARC64
-#define SAVE_SZ 176
-#define SCRATCH_OFF STACK_BIAS + 128
-#define BE_PTR(label) be,pn %xcc, label
-#define SIGN_EXTEND(reg) sra reg, 0, reg
-#else
-#define SAVE_SZ 96
-#define SCRATCH_OFF 72
-#define BE_PTR(label) be label
-#define SIGN_EXTEND(reg)
-#endif
-
-#define SKF_MAX_NEG_OFF (-0x200000) /* SKF_LL_OFF from filter.h */
-
- .text
- .globl bpf_jit_load_word
-bpf_jit_load_word:
- cmp r_OFF, 0
- bl bpf_slow_path_word_neg
- nop
- .globl bpf_jit_load_word_positive_offset
-bpf_jit_load_word_positive_offset:
- sub r_HEADLEN, r_OFF, r_TMP
- cmp r_TMP, 3
- ble bpf_slow_path_word
- add r_SKB_DATA, r_OFF, r_TMP
- andcc r_TMP, 3, %g0
- bne load_word_unaligned
- nop
- retl
- ld [r_TMP], r_A
-load_word_unaligned:
- ldub [r_TMP + 0x0], r_OFF
- ldub [r_TMP + 0x1], r_TMP2
- sll r_OFF, 8, r_OFF
- or r_OFF, r_TMP2, r_OFF
- ldub [r_TMP + 0x2], r_TMP2
- sll r_OFF, 8, r_OFF
- or r_OFF, r_TMP2, r_OFF
- ldub [r_TMP + 0x3], r_TMP2
- sll r_OFF, 8, r_OFF
- retl
- or r_OFF, r_TMP2, r_A
-
- .globl bpf_jit_load_half
-bpf_jit_load_half:
- cmp r_OFF, 0
- bl bpf_slow_path_half_neg
- nop
- .globl bpf_jit_load_half_positive_offset
-bpf_jit_load_half_positive_offset:
- sub r_HEADLEN, r_OFF, r_TMP
- cmp r_TMP, 1
- ble bpf_slow_path_half
- add r_SKB_DATA, r_OFF, r_TMP
- andcc r_TMP, 1, %g0
- bne load_half_unaligned
- nop
- retl
- lduh [r_TMP], r_A
-load_half_unaligned:
- ldub [r_TMP + 0x0], r_OFF
- ldub [r_TMP + 0x1], r_TMP2
- sll r_OFF, 8, r_OFF
- retl
- or r_OFF, r_TMP2, r_A
-
- .globl bpf_jit_load_byte
-bpf_jit_load_byte:
- cmp r_OFF, 0
- bl bpf_slow_path_byte_neg
- nop
- .globl bpf_jit_load_byte_positive_offset
-bpf_jit_load_byte_positive_offset:
- cmp r_OFF, r_HEADLEN
- bge bpf_slow_path_byte
- nop
- retl
- ldub [r_SKB_DATA + r_OFF], r_A
-
- .globl bpf_jit_load_byte_msh
-bpf_jit_load_byte_msh:
- cmp r_OFF, 0
- bl bpf_slow_path_byte_msh_neg
- nop
- .globl bpf_jit_load_byte_msh_positive_offset
-bpf_jit_load_byte_msh_positive_offset:
- cmp r_OFF, r_HEADLEN
- bge bpf_slow_path_byte_msh
- nop
- ldub [r_SKB_DATA + r_OFF], r_OFF
- and r_OFF, 0xf, r_OFF
- retl
- sll r_OFF, 2, r_X
-
-#define bpf_slow_path_common(LEN) \
- save %sp, -SAVE_SZ, %sp; \
- mov %i0, %o0; \
- mov r_OFF, %o1; \
- add %fp, SCRATCH_OFF, %o2; \
- call skb_copy_bits; \
- mov (LEN), %o3; \
- cmp %o0, 0; \
- restore;
-
-bpf_slow_path_word:
- bpf_slow_path_common(4)
- bl bpf_error
- ld [%sp + SCRATCH_OFF], r_A
- retl
- nop
-bpf_slow_path_half:
- bpf_slow_path_common(2)
- bl bpf_error
- lduh [%sp + SCRATCH_OFF], r_A
- retl
- nop
-bpf_slow_path_byte:
- bpf_slow_path_common(1)
- bl bpf_error
- ldub [%sp + SCRATCH_OFF], r_A
- retl
- nop
-bpf_slow_path_byte_msh:
- bpf_slow_path_common(1)
- bl bpf_error
- ldub [%sp + SCRATCH_OFF], r_A
- and r_OFF, 0xf, r_OFF
- retl
- sll r_OFF, 2, r_X
-
-#define bpf_negative_common(LEN) \
- save %sp, -SAVE_SZ, %sp; \
- mov %i0, %o0; \
- mov r_OFF, %o1; \
- SIGN_EXTEND(%o1); \
- call bpf_internal_load_pointer_neg_helper; \
- mov (LEN), %o2; \
- mov %o0, r_TMP; \
- cmp %o0, 0; \
- BE_PTR(bpf_error); \
- restore;
-
-bpf_slow_path_word_neg:
- sethi %hi(SKF_MAX_NEG_OFF), r_TMP
- cmp r_OFF, r_TMP
- bl bpf_error
- nop
- .globl bpf_jit_load_word_negative_offset
-bpf_jit_load_word_negative_offset:
- bpf_negative_common(4)
- andcc r_TMP, 3, %g0
- bne load_word_unaligned
- nop
- retl
- ld [r_TMP], r_A
-
-bpf_slow_path_half_neg:
- sethi %hi(SKF_MAX_NEG_OFF), r_TMP
- cmp r_OFF, r_TMP
- bl bpf_error
- nop
- .globl bpf_jit_load_half_negative_offset
-bpf_jit_load_half_negative_offset:
- bpf_negative_common(2)
- andcc r_TMP, 1, %g0
- bne load_half_unaligned
- nop
- retl
- lduh [r_TMP], r_A
-
-bpf_slow_path_byte_neg:
- sethi %hi(SKF_MAX_NEG_OFF), r_TMP
- cmp r_OFF, r_TMP
- bl bpf_error
- nop
- .globl bpf_jit_load_byte_negative_offset
-bpf_jit_load_byte_negative_offset:
- bpf_negative_common(1)
- retl
- ldub [r_TMP], r_A
-
-bpf_slow_path_byte_msh_neg:
- sethi %hi(SKF_MAX_NEG_OFF), r_TMP
- cmp r_OFF, r_TMP
- bl bpf_error
- nop
- .globl bpf_jit_load_byte_msh_negative_offset
-bpf_jit_load_byte_msh_negative_offset:
- bpf_negative_common(1)
- ldub [r_TMP], r_OFF
- and r_OFF, 0xf, r_OFF
- retl
- sll r_OFF, 2, r_X
-
-bpf_error:
- /* Make the JIT program return zero. The JIT epilogue
- * stores away the original %o7 into r_saved_O7. The
- * normal leaf function return is to use "retl" which
- * would evalute to "jmpl %o7 + 8, %g0" but we want to
- * use the saved value thus the sequence you see here.
- */
- jmpl r_saved_O7 + 8, %g0
- clr %o0
diff --git a/arch/sparc/net/bpf_jit_asm_32.S b/arch/sparc/net/bpf_jit_asm_32.S
new file mode 100644
index 0000000..5632cdc
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_asm_32.S
@@ -0,0 +1,208 @@
+#include <asm/ptrace.h>
+
+#include "bpf_jit_32.h"
+
+#ifdef CONFIG_SPARC64
+#define SAVE_SZ 176
+#define SCRATCH_OFF STACK_BIAS + 128
+#define BE_PTR(label) be,pn %xcc, label
+#define SIGN_EXTEND(reg) sra reg, 0, reg
+#else
+#define SAVE_SZ 96
+#define SCRATCH_OFF 72
+#define BE_PTR(label) be label
+#define SIGN_EXTEND(reg)
+#endif
+
+#define SKF_MAX_NEG_OFF (-0x200000) /* SKF_LL_OFF from filter.h */
+
+ .text
+ .globl bpf_jit_load_word
+bpf_jit_load_word:
+ cmp r_OFF, 0
+ bl bpf_slow_path_word_neg
+ nop
+ .globl bpf_jit_load_word_positive_offset
+bpf_jit_load_word_positive_offset:
+ sub r_HEADLEN, r_OFF, r_TMP
+ cmp r_TMP, 3
+ ble bpf_slow_path_word
+ add r_SKB_DATA, r_OFF, r_TMP
+ andcc r_TMP, 3, %g0
+ bne load_word_unaligned
+ nop
+ retl
+ ld [r_TMP], r_A
+load_word_unaligned:
+ ldub [r_TMP + 0x0], r_OFF
+ ldub [r_TMP + 0x1], r_TMP2
+ sll r_OFF, 8, r_OFF
+ or r_OFF, r_TMP2, r_OFF
+ ldub [r_TMP + 0x2], r_TMP2
+ sll r_OFF, 8, r_OFF
+ or r_OFF, r_TMP2, r_OFF
+ ldub [r_TMP + 0x3], r_TMP2
+ sll r_OFF, 8, r_OFF
+ retl
+ or r_OFF, r_TMP2, r_A
+
+ .globl bpf_jit_load_half
+bpf_jit_load_half:
+ cmp r_OFF, 0
+ bl bpf_slow_path_half_neg
+ nop
+ .globl bpf_jit_load_half_positive_offset
+bpf_jit_load_half_positive_offset:
+ sub r_HEADLEN, r_OFF, r_TMP
+ cmp r_TMP, 1
+ ble bpf_slow_path_half
+ add r_SKB_DATA, r_OFF, r_TMP
+ andcc r_TMP, 1, %g0
+ bne load_half_unaligned
+ nop
+ retl
+ lduh [r_TMP], r_A
+load_half_unaligned:
+ ldub [r_TMP + 0x0], r_OFF
+ ldub [r_TMP + 0x1], r_TMP2
+ sll r_OFF, 8, r_OFF
+ retl
+ or r_OFF, r_TMP2, r_A
+
+ .globl bpf_jit_load_byte
+bpf_jit_load_byte:
+ cmp r_OFF, 0
+ bl bpf_slow_path_byte_neg
+ nop
+ .globl bpf_jit_load_byte_positive_offset
+bpf_jit_load_byte_positive_offset:
+ cmp r_OFF, r_HEADLEN
+ bge bpf_slow_path_byte
+ nop
+ retl
+ ldub [r_SKB_DATA + r_OFF], r_A
+
+ .globl bpf_jit_load_byte_msh
+bpf_jit_load_byte_msh:
+ cmp r_OFF, 0
+ bl bpf_slow_path_byte_msh_neg
+ nop
+ .globl bpf_jit_load_byte_msh_positive_offset
+bpf_jit_load_byte_msh_positive_offset:
+ cmp r_OFF, r_HEADLEN
+ bge bpf_slow_path_byte_msh
+ nop
+ ldub [r_SKB_DATA + r_OFF], r_OFF
+ and r_OFF, 0xf, r_OFF
+ retl
+ sll r_OFF, 2, r_X
+
+#define bpf_slow_path_common(LEN) \
+ save %sp, -SAVE_SZ, %sp; \
+ mov %i0, %o0; \
+ mov r_OFF, %o1; \
+ add %fp, SCRATCH_OFF, %o2; \
+ call skb_copy_bits; \
+ mov (LEN), %o3; \
+ cmp %o0, 0; \
+ restore;
+
+bpf_slow_path_word:
+ bpf_slow_path_common(4)
+ bl bpf_error
+ ld [%sp + SCRATCH_OFF], r_A
+ retl
+ nop
+bpf_slow_path_half:
+ bpf_slow_path_common(2)
+ bl bpf_error
+ lduh [%sp + SCRATCH_OFF], r_A
+ retl
+ nop
+bpf_slow_path_byte:
+ bpf_slow_path_common(1)
+ bl bpf_error
+ ldub [%sp + SCRATCH_OFF], r_A
+ retl
+ nop
+bpf_slow_path_byte_msh:
+ bpf_slow_path_common(1)
+ bl bpf_error
+ ldub [%sp + SCRATCH_OFF], r_A
+ and r_OFF, 0xf, r_OFF
+ retl
+ sll r_OFF, 2, r_X
+
+#define bpf_negative_common(LEN) \
+ save %sp, -SAVE_SZ, %sp; \
+ mov %i0, %o0; \
+ mov r_OFF, %o1; \
+ SIGN_EXTEND(%o1); \
+ call bpf_internal_load_pointer_neg_helper; \
+ mov (LEN), %o2; \
+ mov %o0, r_TMP; \
+ cmp %o0, 0; \
+ BE_PTR(bpf_error); \
+ restore;
+
+bpf_slow_path_word_neg:
+ sethi %hi(SKF_MAX_NEG_OFF), r_TMP
+ cmp r_OFF, r_TMP
+ bl bpf_error
+ nop
+ .globl bpf_jit_load_word_negative_offset
+bpf_jit_load_word_negative_offset:
+ bpf_negative_common(4)
+ andcc r_TMP, 3, %g0
+ bne load_word_unaligned
+ nop
+ retl
+ ld [r_TMP], r_A
+
+bpf_slow_path_half_neg:
+ sethi %hi(SKF_MAX_NEG_OFF), r_TMP
+ cmp r_OFF, r_TMP
+ bl bpf_error
+ nop
+ .globl bpf_jit_load_half_negative_offset
+bpf_jit_load_half_negative_offset:
+ bpf_negative_common(2)
+ andcc r_TMP, 1, %g0
+ bne load_half_unaligned
+ nop
+ retl
+ lduh [r_TMP], r_A
+
+bpf_slow_path_byte_neg:
+ sethi %hi(SKF_MAX_NEG_OFF), r_TMP
+ cmp r_OFF, r_TMP
+ bl bpf_error
+ nop
+ .globl bpf_jit_load_byte_negative_offset
+bpf_jit_load_byte_negative_offset:
+ bpf_negative_common(1)
+ retl
+ ldub [r_TMP], r_A
+
+bpf_slow_path_byte_msh_neg:
+ sethi %hi(SKF_MAX_NEG_OFF), r_TMP
+ cmp r_OFF, r_TMP
+ bl bpf_error
+ nop
+ .globl bpf_jit_load_byte_msh_negative_offset
+bpf_jit_load_byte_msh_negative_offset:
+ bpf_negative_common(1)
+ ldub [r_TMP], r_OFF
+ and r_OFF, 0xf, r_OFF
+ retl
+ sll r_OFF, 2, r_X
+
+bpf_error:
+ /* Make the JIT program return zero. The JIT epilogue
+ * stores away the original %o7 into r_saved_O7. The
+ * normal leaf function return is to use "retl" which
+ * would evalute to "jmpl %o7 + 8, %g0" but we want to
+ * use the saved value thus the sequence you see here.
+ */
+ jmpl r_saved_O7 + 8, %g0
+ clr %o0
diff --git a/arch/sparc/net/bpf_jit_asm_64.S b/arch/sparc/net/bpf_jit_asm_64.S
new file mode 100644
index 0000000..6fb023f
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_asm_64.S
@@ -0,0 +1 @@
+#include "bpf_jit_asm_32.S"
diff --git a/arch/sparc/net/bpf_jit_comp.c b/arch/sparc/net/bpf_jit_comp.c
deleted file mode 100644
index a6d9204..0000000
--- a/arch/sparc/net/bpf_jit_comp.c
+++ /dev/null
@@ -1,815 +0,0 @@
-#include <linux/moduleloader.h>
-#include <linux/workqueue.h>
-#include <linux/netdevice.h>
-#include <linux/filter.h>
-#include <linux/cache.h>
-#include <linux/if_vlan.h>
-
-#include <asm/cacheflush.h>
-#include <asm/ptrace.h>
-
-#include "bpf_jit.h"
-
-int bpf_jit_enable __read_mostly;
-
-static inline bool is_simm13(unsigned int value)
-{
- return value + 0x1000 < 0x2000;
-}
-
-static void bpf_flush_icache(void *start_, void *end_)
-{
-#ifdef CONFIG_SPARC64
- /* Cheetah's I-cache is fully coherent. */
- if (tlb_type == spitfire) {
- unsigned long start = (unsigned long) start_;
- unsigned long end = (unsigned long) end_;
-
- start &= ~7UL;
- end = (end + 7UL) & ~7UL;
- while (start < end) {
- flushi(start);
- start += 32;
- }
- }
-#endif
-}
-
-#define SEEN_DATAREF 1 /* might call external helpers */
-#define SEEN_XREG 2 /* ebx is used */
-#define SEEN_MEM 4 /* use mem[] for temporary storage */
-
-#define S13(X) ((X) & 0x1fff)
-#define IMMED 0x00002000
-#define RD(X) ((X) << 25)
-#define RS1(X) ((X) << 14)
-#define RS2(X) ((X))
-#define OP(X) ((X) << 30)
-#define OP2(X) ((X) << 22)
-#define OP3(X) ((X) << 19)
-#define COND(X) ((X) << 25)
-#define F1(X) OP(X)
-#define F2(X, Y) (OP(X) | OP2(Y))
-#define F3(X, Y) (OP(X) | OP3(Y))
-
-#define CONDN COND(0x0)
-#define CONDE COND(0x1)
-#define CONDLE COND(0x2)
-#define CONDL COND(0x3)
-#define CONDLEU COND(0x4)
-#define CONDCS COND(0x5)
-#define CONDNEG COND(0x6)
-#define CONDVC COND(0x7)
-#define CONDA COND(0x8)
-#define CONDNE COND(0x9)
-#define CONDG COND(0xa)
-#define CONDGE COND(0xb)
-#define CONDGU COND(0xc)
-#define CONDCC COND(0xd)
-#define CONDPOS COND(0xe)
-#define CONDVS COND(0xf)
-
-#define CONDGEU CONDCC
-#define CONDLU CONDCS
-
-#define WDISP22(X) (((X) >> 2) & 0x3fffff)
-
-#define BA (F2(0, 2) | CONDA)
-#define BGU (F2(0, 2) | CONDGU)
-#define BLEU (F2(0, 2) | CONDLEU)
-#define BGEU (F2(0, 2) | CONDGEU)
-#define BLU (F2(0, 2) | CONDLU)
-#define BE (F2(0, 2) | CONDE)
-#define BNE (F2(0, 2) | CONDNE)
-
-#ifdef CONFIG_SPARC64
-#define BE_PTR (F2(0, 1) | CONDE | (2 << 20))
-#else
-#define BE_PTR BE
-#endif
-
-#define SETHI(K, REG) \
- (F2(0, 0x4) | RD(REG) | (((K) >> 10) & 0x3fffff))
-#define OR_LO(K, REG) \
- (F3(2, 0x02) | IMMED | RS1(REG) | ((K) & 0x3ff) | RD(REG))
-
-#define ADD F3(2, 0x00)
-#define AND F3(2, 0x01)
-#define ANDCC F3(2, 0x11)
-#define OR F3(2, 0x02)
-#define XOR F3(2, 0x03)
-#define SUB F3(2, 0x04)
-#define SUBCC F3(2, 0x14)
-#define MUL F3(2, 0x0a) /* umul */
-#define DIV F3(2, 0x0e) /* udiv */
-#define SLL F3(2, 0x25)
-#define SRL F3(2, 0x26)
-#define JMPL F3(2, 0x38)
-#define CALL F1(1)
-#define BR F2(0, 0x01)
-#define RD_Y F3(2, 0x28)
-#define WR_Y F3(2, 0x30)
-
-#define LD32 F3(3, 0x00)
-#define LD8 F3(3, 0x01)
-#define LD16 F3(3, 0x02)
-#define LD64 F3(3, 0x0b)
-#define ST32 F3(3, 0x04)
-
-#ifdef CONFIG_SPARC64
-#define LDPTR LD64
-#define BASE_STACKFRAME 176
-#else
-#define LDPTR LD32
-#define BASE_STACKFRAME 96
-#endif
-
-#define LD32I (LD32 | IMMED)
-#define LD8I (LD8 | IMMED)
-#define LD16I (LD16 | IMMED)
-#define LD64I (LD64 | IMMED)
-#define LDPTRI (LDPTR | IMMED)
-#define ST32I (ST32 | IMMED)
-
-#define emit_nop() \
-do { \
- *prog++ = SETHI(0, G0); \
-} while (0)
-
-#define emit_neg() \
-do { /* sub %g0, r_A, r_A */ \
- *prog++ = SUB | RS1(G0) | RS2(r_A) | RD(r_A); \
-} while (0)
-
-#define emit_reg_move(FROM, TO) \
-do { /* or %g0, FROM, TO */ \
- *prog++ = OR | RS1(G0) | RS2(FROM) | RD(TO); \
-} while (0)
-
-#define emit_clear(REG) \
-do { /* or %g0, %g0, REG */ \
- *prog++ = OR | RS1(G0) | RS2(G0) | RD(REG); \
-} while (0)
-
-#define emit_set_const(K, REG) \
-do { /* sethi %hi(K), REG */ \
- *prog++ = SETHI(K, REG); \
- /* or REG, %lo(K), REG */ \
- *prog++ = OR_LO(K, REG); \
-} while (0)
-
- /* Emit
- *
- * OP r_A, r_X, r_A
- */
-#define emit_alu_X(OPCODE) \
-do { \
- seen |= SEEN_XREG; \
- *prog++ = OPCODE | RS1(r_A) | RS2(r_X) | RD(r_A); \
-} while (0)
-
- /* Emit either:
- *
- * OP r_A, K, r_A
- *
- * or
- *
- * sethi %hi(K), r_TMP
- * or r_TMP, %lo(K), r_TMP
- * OP r_A, r_TMP, r_A
- *
- * depending upon whether K fits in a signed 13-bit
- * immediate instruction field. Emit nothing if K
- * is zero.
- */
-#define emit_alu_K(OPCODE, K) \
-do { \
- if (K || OPCODE == AND || OPCODE == MUL) { \
- unsigned int _insn = OPCODE; \
- _insn |= RS1(r_A) | RD(r_A); \
- if (is_simm13(K)) { \
- *prog++ = _insn | IMMED | S13(K); \
- } else { \
- emit_set_const(K, r_TMP); \
- *prog++ = _insn | RS2(r_TMP); \
- } \
- } \
-} while (0)
-
-#define emit_loadimm(K, DEST) \
-do { \
- if (is_simm13(K)) { \
- /* or %g0, K, DEST */ \
- *prog++ = OR | IMMED | RS1(G0) | S13(K) | RD(DEST); \
- } else { \
- emit_set_const(K, DEST); \
- } \
-} while (0)
-
-#define emit_loadptr(BASE, STRUCT, FIELD, DEST) \
-do { unsigned int _off = offsetof(STRUCT, FIELD); \
- BUILD_BUG_ON(FIELD_SIZEOF(STRUCT, FIELD) != sizeof(void *)); \
- *prog++ = LDPTRI | RS1(BASE) | S13(_off) | RD(DEST); \
-} while (0)
-
-#define emit_load32(BASE, STRUCT, FIELD, DEST) \
-do { unsigned int _off = offsetof(STRUCT, FIELD); \
- BUILD_BUG_ON(FIELD_SIZEOF(STRUCT, FIELD) != sizeof(u32)); \
- *prog++ = LD32I | RS1(BASE) | S13(_off) | RD(DEST); \
-} while (0)
-
-#define emit_load16(BASE, STRUCT, FIELD, DEST) \
-do { unsigned int _off = offsetof(STRUCT, FIELD); \
- BUILD_BUG_ON(FIELD_SIZEOF(STRUCT, FIELD) != sizeof(u16)); \
- *prog++ = LD16I | RS1(BASE) | S13(_off) | RD(DEST); \
-} while (0)
-
-#define __emit_load8(BASE, STRUCT, FIELD, DEST) \
-do { unsigned int _off = offsetof(STRUCT, FIELD); \
- *prog++ = LD8I | RS1(BASE) | S13(_off) | RD(DEST); \
-} while (0)
-
-#define emit_load8(BASE, STRUCT, FIELD, DEST) \
-do { BUILD_BUG_ON(FIELD_SIZEOF(STRUCT, FIELD) != sizeof(u8)); \
- __emit_load8(BASE, STRUCT, FIELD, DEST); \
-} while (0)
-
-#ifdef CONFIG_SPARC64
-#define BIAS (STACK_BIAS - 4)
-#else
-#define BIAS (-4)
-#endif
-
-#define emit_ldmem(OFF, DEST) \
-do { *prog++ = LD32I | RS1(SP) | S13(BIAS - (OFF)) | RD(DEST); \
-} while (0)
-
-#define emit_stmem(OFF, SRC) \
-do { *prog++ = ST32I | RS1(SP) | S13(BIAS - (OFF)) | RD(SRC); \
-} while (0)
-
-#ifdef CONFIG_SMP
-#ifdef CONFIG_SPARC64
-#define emit_load_cpu(REG) \
- emit_load16(G6, struct thread_info, cpu, REG)
-#else
-#define emit_load_cpu(REG) \
- emit_load32(G6, struct thread_info, cpu, REG)
-#endif
-#else
-#define emit_load_cpu(REG) emit_clear(REG)
-#endif
-
-#define emit_skb_loadptr(FIELD, DEST) \
- emit_loadptr(r_SKB, struct sk_buff, FIELD, DEST)
-#define emit_skb_load32(FIELD, DEST) \
- emit_load32(r_SKB, struct sk_buff, FIELD, DEST)
-#define emit_skb_load16(FIELD, DEST) \
- emit_load16(r_SKB, struct sk_buff, FIELD, DEST)
-#define __emit_skb_load8(FIELD, DEST) \
- __emit_load8(r_SKB, struct sk_buff, FIELD, DEST)
-#define emit_skb_load8(FIELD, DEST) \
- emit_load8(r_SKB, struct sk_buff, FIELD, DEST)
-
-#define emit_jmpl(BASE, IMM_OFF, LREG) \
- *prog++ = (JMPL | IMMED | RS1(BASE) | S13(IMM_OFF) | RD(LREG))
-
-#define emit_call(FUNC) \
-do { void *_here = image + addrs[i] - 8; \
- unsigned int _off = (void *)(FUNC) - _here; \
- *prog++ = CALL | (((_off) >> 2) & 0x3fffffff); \
- emit_nop(); \
-} while (0)
-
-#define emit_branch(BR_OPC, DEST) \
-do { unsigned int _here = addrs[i] - 8; \
- *prog++ = BR_OPC | WDISP22((DEST) - _here); \
-} while (0)
-
-#define emit_branch_off(BR_OPC, OFF) \
-do { *prog++ = BR_OPC | WDISP22(OFF); \
-} while (0)
-
-#define emit_jump(DEST) emit_branch(BA, DEST)
-
-#define emit_read_y(REG) *prog++ = RD_Y | RD(REG)
-#define emit_write_y(REG) *prog++ = WR_Y | IMMED | RS1(REG) | S13(0)
-
-#define emit_cmp(R1, R2) \
- *prog++ = (SUBCC | RS1(R1) | RS2(R2) | RD(G0))
-
-#define emit_cmpi(R1, IMM) \
- *prog++ = (SUBCC | IMMED | RS1(R1) | S13(IMM) | RD(G0));
-
-#define emit_btst(R1, R2) \
- *prog++ = (ANDCC | RS1(R1) | RS2(R2) | RD(G0))
-
-#define emit_btsti(R1, IMM) \
- *prog++ = (ANDCC | IMMED | RS1(R1) | S13(IMM) | RD(G0));
-
-#define emit_sub(R1, R2, R3) \
- *prog++ = (SUB | RS1(R1) | RS2(R2) | RD(R3))
-
-#define emit_subi(R1, IMM, R3) \
- *prog++ = (SUB | IMMED | RS1(R1) | S13(IMM) | RD(R3))
-
-#define emit_add(R1, R2, R3) \
- *prog++ = (ADD | RS1(R1) | RS2(R2) | RD(R3))
-
-#define emit_addi(R1, IMM, R3) \
- *prog++ = (ADD | IMMED | RS1(R1) | S13(IMM) | RD(R3))
-
-#define emit_and(R1, R2, R3) \
- *prog++ = (AND | RS1(R1) | RS2(R2) | RD(R3))
-
-#define emit_andi(R1, IMM, R3) \
- *prog++ = (AND | IMMED | RS1(R1) | S13(IMM) | RD(R3))
-
-#define emit_alloc_stack(SZ) \
- *prog++ = (SUB | IMMED | RS1(SP) | S13(SZ) | RD(SP))
-
-#define emit_release_stack(SZ) \
- *prog++ = (ADD | IMMED | RS1(SP) | S13(SZ) | RD(SP))
-
-/* A note about branch offset calculations. The addrs[] array,
- * indexed by BPF instruction, records the address after all the
- * sparc instructions emitted for that BPF instruction.
- *
- * The most common case is to emit a branch at the end of such
- * a code sequence. So this would be two instructions, the
- * branch and it's delay slot.
- *
- * Therefore by default the branch emitters calculate the branch
- * offset field as:
- *
- * destination - (addrs[i] - 8)
- *
- * This "addrs[i] - 8" is the address of the branch itself or
- * what "." would be in assembler notation. The "8" part is
- * how we take into consideration the branch and it's delay
- * slot mentioned above.
- *
- * Sometimes we need to emit a branch earlier in the code
- * sequence. And in these situations we adjust "destination"
- * to accommodate this difference. For example, if we needed
- * to emit a branch (and it's delay slot) right before the
- * final instruction emitted for a BPF opcode, we'd use
- * "destination + 4" instead of just plain "destination" above.
- *
- * This is why you see all of these funny emit_branch() and
- * emit_jump() calls with adjusted offsets.
- */
-
-void bpf_jit_compile(struct bpf_prog *fp)
-{
- unsigned int cleanup_addr, proglen, oldproglen = 0;
- u32 temp[8], *prog, *func, seen = 0, pass;
- const struct sock_filter *filter = fp->insns;
- int i, flen = fp->len, pc_ret0 = -1;
- unsigned int *addrs;
- void *image;
-
- if (!bpf_jit_enable)
- return;
-
- addrs = kmalloc(flen * sizeof(*addrs), GFP_KERNEL);
- if (addrs == NULL)
- return;
-
- /* Before first pass, make a rough estimation of addrs[]
- * each bpf instruction is translated to less than 64 bytes
- */
- for (proglen = 0, i = 0; i < flen; i++) {
- proglen += 64;
- addrs[i] = proglen;
- }
- cleanup_addr = proglen; /* epilogue address */
- image = NULL;
- for (pass = 0; pass < 10; pass++) {
- u8 seen_or_pass0 = (pass == 0) ? (SEEN_XREG | SEEN_DATAREF | SEEN_MEM) : seen;
-
- /* no prologue/epilogue for trivial filters (RET something) */
- proglen = 0;
- prog = temp;
-
- /* Prologue */
- if (seen_or_pass0) {
- if (seen_or_pass0 & SEEN_MEM) {
- unsigned int sz = BASE_STACKFRAME;
- sz += BPF_MEMWORDS * sizeof(u32);
- emit_alloc_stack(sz);
- }
-
- /* Make sure we dont leek kernel memory. */
- if (seen_or_pass0 & SEEN_XREG)
- emit_clear(r_X);
-
- /* If this filter needs to access skb data,
- * load %o4 and %o5 with:
- * %o4 = skb->len - skb->data_len
- * %o5 = skb->data
- * And also back up %o7 into r_saved_O7 so we can
- * invoke the stubs using 'call'.
- */
- if (seen_or_pass0 & SEEN_DATAREF) {
- emit_load32(r_SKB, struct sk_buff, len, r_HEADLEN);
- emit_load32(r_SKB, struct sk_buff, data_len, r_TMP);
- emit_sub(r_HEADLEN, r_TMP, r_HEADLEN);
- emit_loadptr(r_SKB, struct sk_buff, data, r_SKB_DATA);
- }
- }
- emit_reg_move(O7, r_saved_O7);
-
- /* Make sure we dont leak kernel information to the user. */
- if (bpf_needs_clear_a(&filter[0]))
- emit_clear(r_A); /* A = 0 */
-
- for (i = 0; i < flen; i++) {
- unsigned int K = filter[i].k;
- unsigned int t_offset;
- unsigned int f_offset;
- u32 t_op, f_op;
- u16 code = bpf_anc_helper(&filter[i]);
- int ilen;
-
- switch (code) {
- case BPF_ALU | BPF_ADD | BPF_X: /* A += X; */
- emit_alu_X(ADD);
- break;
- case BPF_ALU | BPF_ADD | BPF_K: /* A += K; */
- emit_alu_K(ADD, K);
- break;
- case BPF_ALU | BPF_SUB | BPF_X: /* A -= X; */
- emit_alu_X(SUB);
- break;
- case BPF_ALU | BPF_SUB | BPF_K: /* A -= K */
- emit_alu_K(SUB, K);
- break;
- case BPF_ALU | BPF_AND | BPF_X: /* A &= X */
- emit_alu_X(AND);
- break;
- case BPF_ALU | BPF_AND | BPF_K: /* A &= K */
- emit_alu_K(AND, K);
- break;
- case BPF_ALU | BPF_OR | BPF_X: /* A |= X */
- emit_alu_X(OR);
- break;
- case BPF_ALU | BPF_OR | BPF_K: /* A |= K */
- emit_alu_K(OR, K);
- break;
- case BPF_ANC | SKF_AD_ALU_XOR_X: /* A ^= X; */
- case BPF_ALU | BPF_XOR | BPF_X:
- emit_alu_X(XOR);
- break;
- case BPF_ALU | BPF_XOR | BPF_K: /* A ^= K */
- emit_alu_K(XOR, K);
- break;
- case BPF_ALU | BPF_LSH | BPF_X: /* A <<= X */
- emit_alu_X(SLL);
- break;
- case BPF_ALU | BPF_LSH | BPF_K: /* A <<= K */
- emit_alu_K(SLL, K);
- break;
- case BPF_ALU | BPF_RSH | BPF_X: /* A >>= X */
- emit_alu_X(SRL);
- break;
- case BPF_ALU | BPF_RSH | BPF_K: /* A >>= K */
- emit_alu_K(SRL, K);
- break;
- case BPF_ALU | BPF_MUL | BPF_X: /* A *= X; */
- emit_alu_X(MUL);
- break;
- case BPF_ALU | BPF_MUL | BPF_K: /* A *= K */
- emit_alu_K(MUL, K);
- break;
- case BPF_ALU | BPF_DIV | BPF_K: /* A /= K with K != 0*/
- if (K == 1)
- break;
- emit_write_y(G0);
-#ifdef CONFIG_SPARC32
- /* The Sparc v8 architecture requires
- * three instructions between a %y
- * register write and the first use.
- */
- emit_nop();
- emit_nop();
- emit_nop();
-#endif
- emit_alu_K(DIV, K);
- break;
- case BPF_ALU | BPF_DIV | BPF_X: /* A /= X; */
- emit_cmpi(r_X, 0);
- if (pc_ret0 > 0) {
- t_offset = addrs[pc_ret0 - 1];
-#ifdef CONFIG_SPARC32
- emit_branch(BE, t_offset + 20);
-#else
- emit_branch(BE, t_offset + 8);
-#endif
- emit_nop(); /* delay slot */
- } else {
- emit_branch_off(BNE, 16);
- emit_nop();
-#ifdef CONFIG_SPARC32
- emit_jump(cleanup_addr + 20);
-#else
- emit_jump(cleanup_addr + 8);
-#endif
- emit_clear(r_A);
- }
- emit_write_y(G0);
-#ifdef CONFIG_SPARC32
- /* The Sparc v8 architecture requires
- * three instructions between a %y
- * register write and the first use.
- */
- emit_nop();
- emit_nop();
- emit_nop();
-#endif
- emit_alu_X(DIV);
- break;
- case BPF_ALU | BPF_NEG:
- emit_neg();
- break;
- case BPF_RET | BPF_K:
- if (!K) {
- if (pc_ret0 == -1)
- pc_ret0 = i;
- emit_clear(r_A);
- } else {
- emit_loadimm(K, r_A);
- }
- /* Fallthrough */
- case BPF_RET | BPF_A:
- if (seen_or_pass0) {
- if (i != flen - 1) {
- emit_jump(cleanup_addr);
- emit_nop();
- break;
- }
- if (seen_or_pass0 & SEEN_MEM) {
- unsigned int sz = BASE_STACKFRAME;
- sz += BPF_MEMWORDS * sizeof(u32);
- emit_release_stack(sz);
- }
- }
- /* jmpl %r_saved_O7 + 8, %g0 */
- emit_jmpl(r_saved_O7, 8, G0);
- emit_reg_move(r_A, O0); /* delay slot */
- break;
- case BPF_MISC | BPF_TAX:
- seen |= SEEN_XREG;
- emit_reg_move(r_A, r_X);
- break;
- case BPF_MISC | BPF_TXA:
- seen |= SEEN_XREG;
- emit_reg_move(r_X, r_A);
- break;
- case BPF_ANC | SKF_AD_CPU:
- emit_load_cpu(r_A);
- break;
- case BPF_ANC | SKF_AD_PROTOCOL:
- emit_skb_load16(protocol, r_A);
- break;
- case BPF_ANC | SKF_AD_PKTTYPE:
- __emit_skb_load8(__pkt_type_offset, r_A);
- emit_andi(r_A, PKT_TYPE_MAX, r_A);
- emit_alu_K(SRL, 5);
- break;
- case BPF_ANC | SKF_AD_IFINDEX:
- emit_skb_loadptr(dev, r_A);
- emit_cmpi(r_A, 0);
- emit_branch(BE_PTR, cleanup_addr + 4);
- emit_nop();
- emit_load32(r_A, struct net_device, ifindex, r_A);
- break;
- case BPF_ANC | SKF_AD_MARK:
- emit_skb_load32(mark, r_A);
- break;
- case BPF_ANC | SKF_AD_QUEUE:
- emit_skb_load16(queue_mapping, r_A);
- break;
- case BPF_ANC | SKF_AD_HATYPE:
- emit_skb_loadptr(dev, r_A);
- emit_cmpi(r_A, 0);
- emit_branch(BE_PTR, cleanup_addr + 4);
- emit_nop();
- emit_load16(r_A, struct net_device, type, r_A);
- break;
- case BPF_ANC | SKF_AD_RXHASH:
- emit_skb_load32(hash, r_A);
- break;
- case BPF_ANC | SKF_AD_VLAN_TAG:
- case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
- emit_skb_load16(vlan_tci, r_A);
- if (code != (BPF_ANC | SKF_AD_VLAN_TAG)) {
- emit_alu_K(SRL, 12);
- emit_andi(r_A, 1, r_A);
- } else {
- emit_loadimm(~VLAN_TAG_PRESENT, r_TMP);
- emit_and(r_A, r_TMP, r_A);
- }
- break;
- case BPF_LD | BPF_W | BPF_LEN:
- emit_skb_load32(len, r_A);
- break;
- case BPF_LDX | BPF_W | BPF_LEN:
- emit_skb_load32(len, r_X);
- break;
- case BPF_LD | BPF_IMM:
- emit_loadimm(K, r_A);
- break;
- case BPF_LDX | BPF_IMM:
- emit_loadimm(K, r_X);
- break;
- case BPF_LD | BPF_MEM:
- seen |= SEEN_MEM;
- emit_ldmem(K * 4, r_A);
- break;
- case BPF_LDX | BPF_MEM:
- seen |= SEEN_MEM | SEEN_XREG;
- emit_ldmem(K * 4, r_X);
- break;
- case BPF_ST:
- seen |= SEEN_MEM;
- emit_stmem(K * 4, r_A);
- break;
- case BPF_STX:
- seen |= SEEN_MEM | SEEN_XREG;
- emit_stmem(K * 4, r_X);
- break;
-
-#define CHOOSE_LOAD_FUNC(K, func) \
- ((int)K < 0 ? ((int)K >= SKF_LL_OFF ? func##_negative_offset : func) : func##_positive_offset)
-
- case BPF_LD | BPF_W | BPF_ABS:
- func = CHOOSE_LOAD_FUNC(K, bpf_jit_load_word);
-common_load: seen |= SEEN_DATAREF;
- emit_loadimm(K, r_OFF);
- emit_call(func);
- break;
- case BPF_LD | BPF_H | BPF_ABS:
- func = CHOOSE_LOAD_FUNC(K, bpf_jit_load_half);
- goto common_load;
- case BPF_LD | BPF_B | BPF_ABS:
- func = CHOOSE_LOAD_FUNC(K, bpf_jit_load_byte);
- goto common_load;
- case BPF_LDX | BPF_B | BPF_MSH:
- func = CHOOSE_LOAD_FUNC(K, bpf_jit_load_byte_msh);
- goto common_load;
- case BPF_LD | BPF_W | BPF_IND:
- func = bpf_jit_load_word;
-common_load_ind: seen |= SEEN_DATAREF | SEEN_XREG;
- if (K) {
- if (is_simm13(K)) {
- emit_addi(r_X, K, r_OFF);
- } else {
- emit_loadimm(K, r_TMP);
- emit_add(r_X, r_TMP, r_OFF);
- }
- } else {
- emit_reg_move(r_X, r_OFF);
- }
- emit_call(func);
- break;
- case BPF_LD | BPF_H | BPF_IND:
- func = bpf_jit_load_half;
- goto common_load_ind;
- case BPF_LD | BPF_B | BPF_IND:
- func = bpf_jit_load_byte;
- goto common_load_ind;
- case BPF_JMP | BPF_JA:
- emit_jump(addrs[i + K]);
- emit_nop();
- break;
-
-#define COND_SEL(CODE, TOP, FOP) \
- case CODE: \
- t_op = TOP; \
- f_op = FOP; \
- goto cond_branch
-
- COND_SEL(BPF_JMP | BPF_JGT | BPF_K, BGU, BLEU);
- COND_SEL(BPF_JMP | BPF_JGE | BPF_K, BGEU, BLU);
- COND_SEL(BPF_JMP | BPF_JEQ | BPF_K, BE, BNE);
- COND_SEL(BPF_JMP | BPF_JSET | BPF_K, BNE, BE);
- COND_SEL(BPF_JMP | BPF_JGT | BPF_X, BGU, BLEU);
- COND_SEL(BPF_JMP | BPF_JGE | BPF_X, BGEU, BLU);
- COND_SEL(BPF_JMP | BPF_JEQ | BPF_X, BE, BNE);
- COND_SEL(BPF_JMP | BPF_JSET | BPF_X, BNE, BE);
-
-cond_branch: f_offset = addrs[i + filter[i].jf];
- t_offset = addrs[i + filter[i].jt];
-
- /* same targets, can avoid doing the test :) */
- if (filter[i].jt == filter[i].jf) {
- emit_jump(t_offset);
- emit_nop();
- break;
- }
-
- switch (code) {
- case BPF_JMP | BPF_JGT | BPF_X:
- case BPF_JMP | BPF_JGE | BPF_X:
- case BPF_JMP | BPF_JEQ | BPF_X:
- seen |= SEEN_XREG;
- emit_cmp(r_A, r_X);
- break;
- case BPF_JMP | BPF_JSET | BPF_X:
- seen |= SEEN_XREG;
- emit_btst(r_A, r_X);
- break;
- case BPF_JMP | BPF_JEQ | BPF_K:
- case BPF_JMP | BPF_JGT | BPF_K:
- case BPF_JMP | BPF_JGE | BPF_K:
- if (is_simm13(K)) {
- emit_cmpi(r_A, K);
- } else {
- emit_loadimm(K, r_TMP);
- emit_cmp(r_A, r_TMP);
- }
- break;
- case BPF_JMP | BPF_JSET | BPF_K:
- if (is_simm13(K)) {
- emit_btsti(r_A, K);
- } else {
- emit_loadimm(K, r_TMP);
- emit_btst(r_A, r_TMP);
- }
- break;
- }
- if (filter[i].jt != 0) {
- if (filter[i].jf)
- t_offset += 8;
- emit_branch(t_op, t_offset);
- emit_nop(); /* delay slot */
- if (filter[i].jf) {
- emit_jump(f_offset);
- emit_nop();
- }
- break;
- }
- emit_branch(f_op, f_offset);
- emit_nop(); /* delay slot */
- break;
-
- default:
- /* hmm, too complex filter, give up with jit compiler */
- goto out;
- }
- ilen = (void *) prog - (void *) temp;
- if (image) {
- if (unlikely(proglen + ilen > oldproglen)) {
- pr_err("bpb_jit_compile fatal error\n");
- kfree(addrs);
- module_memfree(image);
- return;
- }
- memcpy(image + proglen, temp, ilen);
- }
- proglen += ilen;
- addrs[i] = proglen;
- prog = temp;
- }
- /* last bpf instruction is always a RET :
- * use it to give the cleanup instruction(s) addr
- */
- cleanup_addr = proglen - 8; /* jmpl; mov r_A,%o0; */
- if (seen_or_pass0 & SEEN_MEM)
- cleanup_addr -= 4; /* add %sp, X, %sp; */
-
- if (image) {
- if (proglen != oldproglen)
- pr_err("bpb_jit_compile proglen=%u != oldproglen=%u\n",
- proglen, oldproglen);
- break;
- }
- if (proglen == oldproglen) {
- image = module_alloc(proglen);
- if (!image)
- goto out;
- }
- oldproglen = proglen;
- }
-
- if (bpf_jit_enable > 1)
- bpf_jit_dump(flen, proglen, pass + 1, image);
-
- if (image) {
- bpf_flush_icache(image, image + proglen);
- fp->bpf_func = (void *)image;
- fp->jited = 1;
- }
-out:
- kfree(addrs);
- return;
-}
-
-void bpf_jit_free(struct bpf_prog *fp)
-{
- if (fp->jited)
- module_memfree(fp->bpf_func);
-
- bpf_prog_unlock_free(fp);
-}
diff --git a/arch/sparc/net/bpf_jit_comp_32.c b/arch/sparc/net/bpf_jit_comp_32.c
new file mode 100644
index 0000000..83fc41d
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_comp_32.c
@@ -0,0 +1,815 @@
+#include <linux/moduleloader.h>
+#include <linux/workqueue.h>
+#include <linux/netdevice.h>
+#include <linux/filter.h>
+#include <linux/cache.h>
+#include <linux/if_vlan.h>
+
+#include <asm/cacheflush.h>
+#include <asm/ptrace.h>
+
+#include "bpf_jit_32.h"
+
+int bpf_jit_enable __read_mostly;
+
+static inline bool is_simm13(unsigned int value)
+{
+ return value + 0x1000 < 0x2000;
+}
+
+static void bpf_flush_icache(void *start_, void *end_)
+{
+#ifdef CONFIG_SPARC64
+ /* Cheetah's I-cache is fully coherent. */
+ if (tlb_type == spitfire) {
+ unsigned long start = (unsigned long) start_;
+ unsigned long end = (unsigned long) end_;
+
+ start &= ~7UL;
+ end = (end + 7UL) & ~7UL;
+ while (start < end) {
+ flushi(start);
+ start += 32;
+ }
+ }
+#endif
+}
+
+#define SEEN_DATAREF 1 /* might call external helpers */
+#define SEEN_XREG 2 /* ebx is used */
+#define SEEN_MEM 4 /* use mem[] for temporary storage */
+
+#define S13(X) ((X) & 0x1fff)
+#define IMMED 0x00002000
+#define RD(X) ((X) << 25)
+#define RS1(X) ((X) << 14)
+#define RS2(X) ((X))
+#define OP(X) ((X) << 30)
+#define OP2(X) ((X) << 22)
+#define OP3(X) ((X) << 19)
+#define COND(X) ((X) << 25)
+#define F1(X) OP(X)
+#define F2(X, Y) (OP(X) | OP2(Y))
+#define F3(X, Y) (OP(X) | OP3(Y))
+
+#define CONDN COND(0x0)
+#define CONDE COND(0x1)
+#define CONDLE COND(0x2)
+#define CONDL COND(0x3)
+#define CONDLEU COND(0x4)
+#define CONDCS COND(0x5)
+#define CONDNEG COND(0x6)
+#define CONDVC COND(0x7)
+#define CONDA COND(0x8)
+#define CONDNE COND(0x9)
+#define CONDG COND(0xa)
+#define CONDGE COND(0xb)
+#define CONDGU COND(0xc)
+#define CONDCC COND(0xd)
+#define CONDPOS COND(0xe)
+#define CONDVS COND(0xf)
+
+#define CONDGEU CONDCC
+#define CONDLU CONDCS
+
+#define WDISP22(X) (((X) >> 2) & 0x3fffff)
+
+#define BA (F2(0, 2) | CONDA)
+#define BGU (F2(0, 2) | CONDGU)
+#define BLEU (F2(0, 2) | CONDLEU)
+#define BGEU (F2(0, 2) | CONDGEU)
+#define BLU (F2(0, 2) | CONDLU)
+#define BE (F2(0, 2) | CONDE)
+#define BNE (F2(0, 2) | CONDNE)
+
+#ifdef CONFIG_SPARC64
+#define BE_PTR (F2(0, 1) | CONDE | (2 << 20))
+#else
+#define BE_PTR BE
+#endif
+
+#define SETHI(K, REG) \
+ (F2(0, 0x4) | RD(REG) | (((K) >> 10) & 0x3fffff))
+#define OR_LO(K, REG) \
+ (F3(2, 0x02) | IMMED | RS1(REG) | ((K) & 0x3ff) | RD(REG))
+
+#define ADD F3(2, 0x00)
+#define AND F3(2, 0x01)
+#define ANDCC F3(2, 0x11)
+#define OR F3(2, 0x02)
+#define XOR F3(2, 0x03)
+#define SUB F3(2, 0x04)
+#define SUBCC F3(2, 0x14)
+#define MUL F3(2, 0x0a) /* umul */
+#define DIV F3(2, 0x0e) /* udiv */
+#define SLL F3(2, 0x25)
+#define SRL F3(2, 0x26)
+#define JMPL F3(2, 0x38)
+#define CALL F1(1)
+#define BR F2(0, 0x01)
+#define RD_Y F3(2, 0x28)
+#define WR_Y F3(2, 0x30)
+
+#define LD32 F3(3, 0x00)
+#define LD8 F3(3, 0x01)
+#define LD16 F3(3, 0x02)
+#define LD64 F3(3, 0x0b)
+#define ST32 F3(3, 0x04)
+
+#ifdef CONFIG_SPARC64
+#define LDPTR LD64
+#define BASE_STACKFRAME 176
+#else
+#define LDPTR LD32
+#define BASE_STACKFRAME 96
+#endif
+
+#define LD32I (LD32 | IMMED)
+#define LD8I (LD8 | IMMED)
+#define LD16I (LD16 | IMMED)
+#define LD64I (LD64 | IMMED)
+#define LDPTRI (LDPTR | IMMED)
+#define ST32I (ST32 | IMMED)
+
+#define emit_nop() \
+do { \
+ *prog++ = SETHI(0, G0); \
+} while (0)
+
+#define emit_neg() \
+do { /* sub %g0, r_A, r_A */ \
+ *prog++ = SUB | RS1(G0) | RS2(r_A) | RD(r_A); \
+} while (0)
+
+#define emit_reg_move(FROM, TO) \
+do { /* or %g0, FROM, TO */ \
+ *prog++ = OR | RS1(G0) | RS2(FROM) | RD(TO); \
+} while (0)
+
+#define emit_clear(REG) \
+do { /* or %g0, %g0, REG */ \
+ *prog++ = OR | RS1(G0) | RS2(G0) | RD(REG); \
+} while (0)
+
+#define emit_set_const(K, REG) \
+do { /* sethi %hi(K), REG */ \
+ *prog++ = SETHI(K, REG); \
+ /* or REG, %lo(K), REG */ \
+ *prog++ = OR_LO(K, REG); \
+} while (0)
+
+ /* Emit
+ *
+ * OP r_A, r_X, r_A
+ */
+#define emit_alu_X(OPCODE) \
+do { \
+ seen |= SEEN_XREG; \
+ *prog++ = OPCODE | RS1(r_A) | RS2(r_X) | RD(r_A); \
+} while (0)
+
+ /* Emit either:
+ *
+ * OP r_A, K, r_A
+ *
+ * or
+ *
+ * sethi %hi(K), r_TMP
+ * or r_TMP, %lo(K), r_TMP
+ * OP r_A, r_TMP, r_A
+ *
+ * depending upon whether K fits in a signed 13-bit
+ * immediate instruction field. Emit nothing if K
+ * is zero.
+ */
+#define emit_alu_K(OPCODE, K) \
+do { \
+ if (K || OPCODE == AND || OPCODE == MUL) { \
+ unsigned int _insn = OPCODE; \
+ _insn |= RS1(r_A) | RD(r_A); \
+ if (is_simm13(K)) { \
+ *prog++ = _insn | IMMED | S13(K); \
+ } else { \
+ emit_set_const(K, r_TMP); \
+ *prog++ = _insn | RS2(r_TMP); \
+ } \
+ } \
+} while (0)
+
+#define emit_loadimm(K, DEST) \
+do { \
+ if (is_simm13(K)) { \
+ /* or %g0, K, DEST */ \
+ *prog++ = OR | IMMED | RS1(G0) | S13(K) | RD(DEST); \
+ } else { \
+ emit_set_const(K, DEST); \
+ } \
+} while (0)
+
+#define emit_loadptr(BASE, STRUCT, FIELD, DEST) \
+do { unsigned int _off = offsetof(STRUCT, FIELD); \
+ BUILD_BUG_ON(FIELD_SIZEOF(STRUCT, FIELD) != sizeof(void *)); \
+ *prog++ = LDPTRI | RS1(BASE) | S13(_off) | RD(DEST); \
+} while (0)
+
+#define emit_load32(BASE, STRUCT, FIELD, DEST) \
+do { unsigned int _off = offsetof(STRUCT, FIELD); \
+ BUILD_BUG_ON(FIELD_SIZEOF(STRUCT, FIELD) != sizeof(u32)); \
+ *prog++ = LD32I | RS1(BASE) | S13(_off) | RD(DEST); \
+} while (0)
+
+#define emit_load16(BASE, STRUCT, FIELD, DEST) \
+do { unsigned int _off = offsetof(STRUCT, FIELD); \
+ BUILD_BUG_ON(FIELD_SIZEOF(STRUCT, FIELD) != sizeof(u16)); \
+ *prog++ = LD16I | RS1(BASE) | S13(_off) | RD(DEST); \
+} while (0)
+
+#define __emit_load8(BASE, STRUCT, FIELD, DEST) \
+do { unsigned int _off = offsetof(STRUCT, FIELD); \
+ *prog++ = LD8I | RS1(BASE) | S13(_off) | RD(DEST); \
+} while (0)
+
+#define emit_load8(BASE, STRUCT, FIELD, DEST) \
+do { BUILD_BUG_ON(FIELD_SIZEOF(STRUCT, FIELD) != sizeof(u8)); \
+ __emit_load8(BASE, STRUCT, FIELD, DEST); \
+} while (0)
+
+#ifdef CONFIG_SPARC64
+#define BIAS (STACK_BIAS - 4)
+#else
+#define BIAS (-4)
+#endif
+
+#define emit_ldmem(OFF, DEST) \
+do { *prog++ = LD32I | RS1(SP) | S13(BIAS - (OFF)) | RD(DEST); \
+} while (0)
+
+#define emit_stmem(OFF, SRC) \
+do { *prog++ = ST32I | RS1(SP) | S13(BIAS - (OFF)) | RD(SRC); \
+} while (0)
+
+#ifdef CONFIG_SMP
+#ifdef CONFIG_SPARC64
+#define emit_load_cpu(REG) \
+ emit_load16(G6, struct thread_info, cpu, REG)
+#else
+#define emit_load_cpu(REG) \
+ emit_load32(G6, struct thread_info, cpu, REG)
+#endif
+#else
+#define emit_load_cpu(REG) emit_clear(REG)
+#endif
+
+#define emit_skb_loadptr(FIELD, DEST) \
+ emit_loadptr(r_SKB, struct sk_buff, FIELD, DEST)
+#define emit_skb_load32(FIELD, DEST) \
+ emit_load32(r_SKB, struct sk_buff, FIELD, DEST)
+#define emit_skb_load16(FIELD, DEST) \
+ emit_load16(r_SKB, struct sk_buff, FIELD, DEST)
+#define __emit_skb_load8(FIELD, DEST) \
+ __emit_load8(r_SKB, struct sk_buff, FIELD, DEST)
+#define emit_skb_load8(FIELD, DEST) \
+ emit_load8(r_SKB, struct sk_buff, FIELD, DEST)
+
+#define emit_jmpl(BASE, IMM_OFF, LREG) \
+ *prog++ = (JMPL | IMMED | RS1(BASE) | S13(IMM_OFF) | RD(LREG))
+
+#define emit_call(FUNC) \
+do { void *_here = image + addrs[i] - 8; \
+ unsigned int _off = (void *)(FUNC) - _here; \
+ *prog++ = CALL | (((_off) >> 2) & 0x3fffffff); \
+ emit_nop(); \
+} while (0)
+
+#define emit_branch(BR_OPC, DEST) \
+do { unsigned int _here = addrs[i] - 8; \
+ *prog++ = BR_OPC | WDISP22((DEST) - _here); \
+} while (0)
+
+#define emit_branch_off(BR_OPC, OFF) \
+do { *prog++ = BR_OPC | WDISP22(OFF); \
+} while (0)
+
+#define emit_jump(DEST) emit_branch(BA, DEST)
+
+#define emit_read_y(REG) *prog++ = RD_Y | RD(REG)
+#define emit_write_y(REG) *prog++ = WR_Y | IMMED | RS1(REG) | S13(0)
+
+#define emit_cmp(R1, R2) \
+ *prog++ = (SUBCC | RS1(R1) | RS2(R2) | RD(G0))
+
+#define emit_cmpi(R1, IMM) \
+ *prog++ = (SUBCC | IMMED | RS1(R1) | S13(IMM) | RD(G0));
+
+#define emit_btst(R1, R2) \
+ *prog++ = (ANDCC | RS1(R1) | RS2(R2) | RD(G0))
+
+#define emit_btsti(R1, IMM) \
+ *prog++ = (ANDCC | IMMED | RS1(R1) | S13(IMM) | RD(G0));
+
+#define emit_sub(R1, R2, R3) \
+ *prog++ = (SUB | RS1(R1) | RS2(R2) | RD(R3))
+
+#define emit_subi(R1, IMM, R3) \
+ *prog++ = (SUB | IMMED | RS1(R1) | S13(IMM) | RD(R3))
+
+#define emit_add(R1, R2, R3) \
+ *prog++ = (ADD | RS1(R1) | RS2(R2) | RD(R3))
+
+#define emit_addi(R1, IMM, R3) \
+ *prog++ = (ADD | IMMED | RS1(R1) | S13(IMM) | RD(R3))
+
+#define emit_and(R1, R2, R3) \
+ *prog++ = (AND | RS1(R1) | RS2(R2) | RD(R3))
+
+#define emit_andi(R1, IMM, R3) \
+ *prog++ = (AND | IMMED | RS1(R1) | S13(IMM) | RD(R3))
+
+#define emit_alloc_stack(SZ) \
+ *prog++ = (SUB | IMMED | RS1(SP) | S13(SZ) | RD(SP))
+
+#define emit_release_stack(SZ) \
+ *prog++ = (ADD | IMMED | RS1(SP) | S13(SZ) | RD(SP))
+
+/* A note about branch offset calculations. The addrs[] array,
+ * indexed by BPF instruction, records the address after all the
+ * sparc instructions emitted for that BPF instruction.
+ *
+ * The most common case is to emit a branch at the end of such
+ * a code sequence. So this would be two instructions, the
+ * branch and it's delay slot.
+ *
+ * Therefore by default the branch emitters calculate the branch
+ * offset field as:
+ *
+ * destination - (addrs[i] - 8)
+ *
+ * This "addrs[i] - 8" is the address of the branch itself or
+ * what "." would be in assembler notation. The "8" part is
+ * how we take into consideration the branch and it's delay
+ * slot mentioned above.
+ *
+ * Sometimes we need to emit a branch earlier in the code
+ * sequence. And in these situations we adjust "destination"
+ * to accommodate this difference. For example, if we needed
+ * to emit a branch (and it's delay slot) right before the
+ * final instruction emitted for a BPF opcode, we'd use
+ * "destination + 4" instead of just plain "destination" above.
+ *
+ * This is why you see all of these funny emit_branch() and
+ * emit_jump() calls with adjusted offsets.
+ */
+
+void bpf_jit_compile(struct bpf_prog *fp)
+{
+ unsigned int cleanup_addr, proglen, oldproglen = 0;
+ u32 temp[8], *prog, *func, seen = 0, pass;
+ const struct sock_filter *filter = fp->insns;
+ int i, flen = fp->len, pc_ret0 = -1;
+ unsigned int *addrs;
+ void *image;
+
+ if (!bpf_jit_enable)
+ return;
+
+ addrs = kmalloc(flen * sizeof(*addrs), GFP_KERNEL);
+ if (addrs == NULL)
+ return;
+
+ /* Before first pass, make a rough estimation of addrs[]
+ * each bpf instruction is translated to less than 64 bytes
+ */
+ for (proglen = 0, i = 0; i < flen; i++) {
+ proglen += 64;
+ addrs[i] = proglen;
+ }
+ cleanup_addr = proglen; /* epilogue address */
+ image = NULL;
+ for (pass = 0; pass < 10; pass++) {
+ u8 seen_or_pass0 = (pass == 0) ? (SEEN_XREG | SEEN_DATAREF | SEEN_MEM) : seen;
+
+ /* no prologue/epilogue for trivial filters (RET something) */
+ proglen = 0;
+ prog = temp;
+
+ /* Prologue */
+ if (seen_or_pass0) {
+ if (seen_or_pass0 & SEEN_MEM) {
+ unsigned int sz = BASE_STACKFRAME;
+ sz += BPF_MEMWORDS * sizeof(u32);
+ emit_alloc_stack(sz);
+ }
+
+ /* Make sure we dont leek kernel memory. */
+ if (seen_or_pass0 & SEEN_XREG)
+ emit_clear(r_X);
+
+ /* If this filter needs to access skb data,
+ * load %o4 and %o5 with:
+ * %o4 = skb->len - skb->data_len
+ * %o5 = skb->data
+ * And also back up %o7 into r_saved_O7 so we can
+ * invoke the stubs using 'call'.
+ */
+ if (seen_or_pass0 & SEEN_DATAREF) {
+ emit_load32(r_SKB, struct sk_buff, len, r_HEADLEN);
+ emit_load32(r_SKB, struct sk_buff, data_len, r_TMP);
+ emit_sub(r_HEADLEN, r_TMP, r_HEADLEN);
+ emit_loadptr(r_SKB, struct sk_buff, data, r_SKB_DATA);
+ }
+ }
+ emit_reg_move(O7, r_saved_O7);
+
+ /* Make sure we dont leak kernel information to the user. */
+ if (bpf_needs_clear_a(&filter[0]))
+ emit_clear(r_A); /* A = 0 */
+
+ for (i = 0; i < flen; i++) {
+ unsigned int K = filter[i].k;
+ unsigned int t_offset;
+ unsigned int f_offset;
+ u32 t_op, f_op;
+ u16 code = bpf_anc_helper(&filter[i]);
+ int ilen;
+
+ switch (code) {
+ case BPF_ALU | BPF_ADD | BPF_X: /* A += X; */
+ emit_alu_X(ADD);
+ break;
+ case BPF_ALU | BPF_ADD | BPF_K: /* A += K; */
+ emit_alu_K(ADD, K);
+ break;
+ case BPF_ALU | BPF_SUB | BPF_X: /* A -= X; */
+ emit_alu_X(SUB);
+ break;
+ case BPF_ALU | BPF_SUB | BPF_K: /* A -= K */
+ emit_alu_K(SUB, K);
+ break;
+ case BPF_ALU | BPF_AND | BPF_X: /* A &= X */
+ emit_alu_X(AND);
+ break;
+ case BPF_ALU | BPF_AND | BPF_K: /* A &= K */
+ emit_alu_K(AND, K);
+ break;
+ case BPF_ALU | BPF_OR | BPF_X: /* A |= X */
+ emit_alu_X(OR);
+ break;
+ case BPF_ALU | BPF_OR | BPF_K: /* A |= K */
+ emit_alu_K(OR, K);
+ break;
+ case BPF_ANC | SKF_AD_ALU_XOR_X: /* A ^= X; */
+ case BPF_ALU | BPF_XOR | BPF_X:
+ emit_alu_X(XOR);
+ break;
+ case BPF_ALU | BPF_XOR | BPF_K: /* A ^= K */
+ emit_alu_K(XOR, K);
+ break;
+ case BPF_ALU | BPF_LSH | BPF_X: /* A <<= X */
+ emit_alu_X(SLL);
+ break;
+ case BPF_ALU | BPF_LSH | BPF_K: /* A <<= K */
+ emit_alu_K(SLL, K);
+ break;
+ case BPF_ALU | BPF_RSH | BPF_X: /* A >>= X */
+ emit_alu_X(SRL);
+ break;
+ case BPF_ALU | BPF_RSH | BPF_K: /* A >>= K */
+ emit_alu_K(SRL, K);
+ break;
+ case BPF_ALU | BPF_MUL | BPF_X: /* A *= X; */
+ emit_alu_X(MUL);
+ break;
+ case BPF_ALU | BPF_MUL | BPF_K: /* A *= K */
+ emit_alu_K(MUL, K);
+ break;
+ case BPF_ALU | BPF_DIV | BPF_K: /* A /= K with K != 0*/
+ if (K == 1)
+ break;
+ emit_write_y(G0);
+#ifdef CONFIG_SPARC32
+ /* The Sparc v8 architecture requires
+ * three instructions between a %y
+ * register write and the first use.
+ */
+ emit_nop();
+ emit_nop();
+ emit_nop();
+#endif
+ emit_alu_K(DIV, K);
+ break;
+ case BPF_ALU | BPF_DIV | BPF_X: /* A /= X; */
+ emit_cmpi(r_X, 0);
+ if (pc_ret0 > 0) {
+ t_offset = addrs[pc_ret0 - 1];
+#ifdef CONFIG_SPARC32
+ emit_branch(BE, t_offset + 20);
+#else
+ emit_branch(BE, t_offset + 8);
+#endif
+ emit_nop(); /* delay slot */
+ } else {
+ emit_branch_off(BNE, 16);
+ emit_nop();
+#ifdef CONFIG_SPARC32
+ emit_jump(cleanup_addr + 20);
+#else
+ emit_jump(cleanup_addr + 8);
+#endif
+ emit_clear(r_A);
+ }
+ emit_write_y(G0);
+#ifdef CONFIG_SPARC32
+ /* The Sparc v8 architecture requires
+ * three instructions between a %y
+ * register write and the first use.
+ */
+ emit_nop();
+ emit_nop();
+ emit_nop();
+#endif
+ emit_alu_X(DIV);
+ break;
+ case BPF_ALU | BPF_NEG:
+ emit_neg();
+ break;
+ case BPF_RET | BPF_K:
+ if (!K) {
+ if (pc_ret0 == -1)
+ pc_ret0 = i;
+ emit_clear(r_A);
+ } else {
+ emit_loadimm(K, r_A);
+ }
+ /* Fallthrough */
+ case BPF_RET | BPF_A:
+ if (seen_or_pass0) {
+ if (i != flen - 1) {
+ emit_jump(cleanup_addr);
+ emit_nop();
+ break;
+ }
+ if (seen_or_pass0 & SEEN_MEM) {
+ unsigned int sz = BASE_STACKFRAME;
+ sz += BPF_MEMWORDS * sizeof(u32);
+ emit_release_stack(sz);
+ }
+ }
+ /* jmpl %r_saved_O7 + 8, %g0 */
+ emit_jmpl(r_saved_O7, 8, G0);
+ emit_reg_move(r_A, O0); /* delay slot */
+ break;
+ case BPF_MISC | BPF_TAX:
+ seen |= SEEN_XREG;
+ emit_reg_move(r_A, r_X);
+ break;
+ case BPF_MISC | BPF_TXA:
+ seen |= SEEN_XREG;
+ emit_reg_move(r_X, r_A);
+ break;
+ case BPF_ANC | SKF_AD_CPU:
+ emit_load_cpu(r_A);
+ break;
+ case BPF_ANC | SKF_AD_PROTOCOL:
+ emit_skb_load16(protocol, r_A);
+ break;
+ case BPF_ANC | SKF_AD_PKTTYPE:
+ __emit_skb_load8(__pkt_type_offset, r_A);
+ emit_andi(r_A, PKT_TYPE_MAX, r_A);
+ emit_alu_K(SRL, 5);
+ break;
+ case BPF_ANC | SKF_AD_IFINDEX:
+ emit_skb_loadptr(dev, r_A);
+ emit_cmpi(r_A, 0);
+ emit_branch(BE_PTR, cleanup_addr + 4);
+ emit_nop();
+ emit_load32(r_A, struct net_device, ifindex, r_A);
+ break;
+ case BPF_ANC | SKF_AD_MARK:
+ emit_skb_load32(mark, r_A);
+ break;
+ case BPF_ANC | SKF_AD_QUEUE:
+ emit_skb_load16(queue_mapping, r_A);
+ break;
+ case BPF_ANC | SKF_AD_HATYPE:
+ emit_skb_loadptr(dev, r_A);
+ emit_cmpi(r_A, 0);
+ emit_branch(BE_PTR, cleanup_addr + 4);
+ emit_nop();
+ emit_load16(r_A, struct net_device, type, r_A);
+ break;
+ case BPF_ANC | SKF_AD_RXHASH:
+ emit_skb_load32(hash, r_A);
+ break;
+ case BPF_ANC | SKF_AD_VLAN_TAG:
+ case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
+ emit_skb_load16(vlan_tci, r_A);
+ if (code != (BPF_ANC | SKF_AD_VLAN_TAG)) {
+ emit_alu_K(SRL, 12);
+ emit_andi(r_A, 1, r_A);
+ } else {
+ emit_loadimm(~VLAN_TAG_PRESENT, r_TMP);
+ emit_and(r_A, r_TMP, r_A);
+ }
+ break;
+ case BPF_LD | BPF_W | BPF_LEN:
+ emit_skb_load32(len, r_A);
+ break;
+ case BPF_LDX | BPF_W | BPF_LEN:
+ emit_skb_load32(len, r_X);
+ break;
+ case BPF_LD | BPF_IMM:
+ emit_loadimm(K, r_A);
+ break;
+ case BPF_LDX | BPF_IMM:
+ emit_loadimm(K, r_X);
+ break;
+ case BPF_LD | BPF_MEM:
+ seen |= SEEN_MEM;
+ emit_ldmem(K * 4, r_A);
+ break;
+ case BPF_LDX | BPF_MEM:
+ seen |= SEEN_MEM | SEEN_XREG;
+ emit_ldmem(K * 4, r_X);
+ break;
+ case BPF_ST:
+ seen |= SEEN_MEM;
+ emit_stmem(K * 4, r_A);
+ break;
+ case BPF_STX:
+ seen |= SEEN_MEM | SEEN_XREG;
+ emit_stmem(K * 4, r_X);
+ break;
+
+#define CHOOSE_LOAD_FUNC(K, func) \
+ ((int)K < 0 ? ((int)K >= SKF_LL_OFF ? func##_negative_offset : func) : func##_positive_offset)
+
+ case BPF_LD | BPF_W | BPF_ABS:
+ func = CHOOSE_LOAD_FUNC(K, bpf_jit_load_word);
+common_load: seen |= SEEN_DATAREF;
+ emit_loadimm(K, r_OFF);
+ emit_call(func);
+ break;
+ case BPF_LD | BPF_H | BPF_ABS:
+ func = CHOOSE_LOAD_FUNC(K, bpf_jit_load_half);
+ goto common_load;
+ case BPF_LD | BPF_B | BPF_ABS:
+ func = CHOOSE_LOAD_FUNC(K, bpf_jit_load_byte);
+ goto common_load;
+ case BPF_LDX | BPF_B | BPF_MSH:
+ func = CHOOSE_LOAD_FUNC(K, bpf_jit_load_byte_msh);
+ goto common_load;
+ case BPF_LD | BPF_W | BPF_IND:
+ func = bpf_jit_load_word;
+common_load_ind: seen |= SEEN_DATAREF | SEEN_XREG;
+ if (K) {
+ if (is_simm13(K)) {
+ emit_addi(r_X, K, r_OFF);
+ } else {
+ emit_loadimm(K, r_TMP);
+ emit_add(r_X, r_TMP, r_OFF);
+ }
+ } else {
+ emit_reg_move(r_X, r_OFF);
+ }
+ emit_call(func);
+ break;
+ case BPF_LD | BPF_H | BPF_IND:
+ func = bpf_jit_load_half;
+ goto common_load_ind;
+ case BPF_LD | BPF_B | BPF_IND:
+ func = bpf_jit_load_byte;
+ goto common_load_ind;
+ case BPF_JMP | BPF_JA:
+ emit_jump(addrs[i + K]);
+ emit_nop();
+ break;
+
+#define COND_SEL(CODE, TOP, FOP) \
+ case CODE: \
+ t_op = TOP; \
+ f_op = FOP; \
+ goto cond_branch
+
+ COND_SEL(BPF_JMP | BPF_JGT | BPF_K, BGU, BLEU);
+ COND_SEL(BPF_JMP | BPF_JGE | BPF_K, BGEU, BLU);
+ COND_SEL(BPF_JMP | BPF_JEQ | BPF_K, BE, BNE);
+ COND_SEL(BPF_JMP | BPF_JSET | BPF_K, BNE, BE);
+ COND_SEL(BPF_JMP | BPF_JGT | BPF_X, BGU, BLEU);
+ COND_SEL(BPF_JMP | BPF_JGE | BPF_X, BGEU, BLU);
+ COND_SEL(BPF_JMP | BPF_JEQ | BPF_X, BE, BNE);
+ COND_SEL(BPF_JMP | BPF_JSET | BPF_X, BNE, BE);
+
+cond_branch: f_offset = addrs[i + filter[i].jf];
+ t_offset = addrs[i + filter[i].jt];
+
+ /* same targets, can avoid doing the test :) */
+ if (filter[i].jt == filter[i].jf) {
+ emit_jump(t_offset);
+ emit_nop();
+ break;
+ }
+
+ switch (code) {
+ case BPF_JMP | BPF_JGT | BPF_X:
+ case BPF_JMP | BPF_JGE | BPF_X:
+ case BPF_JMP | BPF_JEQ | BPF_X:
+ seen |= SEEN_XREG;
+ emit_cmp(r_A, r_X);
+ break;
+ case BPF_JMP | BPF_JSET | BPF_X:
+ seen |= SEEN_XREG;
+ emit_btst(r_A, r_X);
+ break;
+ case BPF_JMP | BPF_JEQ | BPF_K:
+ case BPF_JMP | BPF_JGT | BPF_K:
+ case BPF_JMP | BPF_JGE | BPF_K:
+ if (is_simm13(K)) {
+ emit_cmpi(r_A, K);
+ } else {
+ emit_loadimm(K, r_TMP);
+ emit_cmp(r_A, r_TMP);
+ }
+ break;
+ case BPF_JMP | BPF_JSET | BPF_K:
+ if (is_simm13(K)) {
+ emit_btsti(r_A, K);
+ } else {
+ emit_loadimm(K, r_TMP);
+ emit_btst(r_A, r_TMP);
+ }
+ break;
+ }
+ if (filter[i].jt != 0) {
+ if (filter[i].jf)
+ t_offset += 8;
+ emit_branch(t_op, t_offset);
+ emit_nop(); /* delay slot */
+ if (filter[i].jf) {
+ emit_jump(f_offset);
+ emit_nop();
+ }
+ break;
+ }
+ emit_branch(f_op, f_offset);
+ emit_nop(); /* delay slot */
+ break;
+
+ default:
+ /* hmm, too complex filter, give up with jit compiler */
+ goto out;
+ }
+ ilen = (void *) prog - (void *) temp;
+ if (image) {
+ if (unlikely(proglen + ilen > oldproglen)) {
+ pr_err("bpb_jit_compile fatal error\n");
+ kfree(addrs);
+ module_memfree(image);
+ return;
+ }
+ memcpy(image + proglen, temp, ilen);
+ }
+ proglen += ilen;
+ addrs[i] = proglen;
+ prog = temp;
+ }
+ /* last bpf instruction is always a RET :
+ * use it to give the cleanup instruction(s) addr
+ */
+ cleanup_addr = proglen - 8; /* jmpl; mov r_A,%o0; */
+ if (seen_or_pass0 & SEEN_MEM)
+ cleanup_addr -= 4; /* add %sp, X, %sp; */
+
+ if (image) {
+ if (proglen != oldproglen)
+ pr_err("bpb_jit_compile proglen=%u != oldproglen=%u\n",
+ proglen, oldproglen);
+ break;
+ }
+ if (proglen == oldproglen) {
+ image = module_alloc(proglen);
+ if (!image)
+ goto out;
+ }
+ oldproglen = proglen;
+ }
+
+ if (bpf_jit_enable > 1)
+ bpf_jit_dump(flen, proglen, pass + 1, image);
+
+ if (image) {
+ bpf_flush_icache(image, image + proglen);
+ fp->bpf_func = (void *)image;
+ fp->jited = 1;
+ }
+out:
+ kfree(addrs);
+ return;
+}
+
+void bpf_jit_free(struct bpf_prog *fp)
+{
+ if (fp->jited)
+ module_memfree(fp->bpf_func);
+
+ bpf_prog_unlock_free(fp);
+}
diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c
new file mode 100644
index 0000000..49b5f65
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_comp_64.c
@@ -0,0 +1 @@
+#include "bpf_jit_comp_32.c"
--
2.1.2.532.g19b5d50
^ permalink raw reply related
* [PATCH 2/2] sparc64: Add eBPF JIT.
From: David Miller @ 2017-04-18 18:58 UTC (permalink / raw)
To: sparclinux; +Cc: netdev, ast, daniel
This is an eBPF JIT for sparc64. All major features are supported
except for tail calls.
test_bpf passes with no failures and all tests are JIT'd, both with
and without hardening enabled.
Signed-off-by: David S. Miller <davem@davemloft.net>
---
arch/sparc/Kconfig | 3 +-
arch/sparc/net/bpf_jit_64.h | 79 +++
arch/sparc/net/bpf_jit_asm_32.S | 7 -
arch/sparc/net/bpf_jit_asm_64.S | 162 +++++-
arch/sparc/net/bpf_jit_comp_32.c | 49 --
arch/sparc/net/bpf_jit_comp_64.c | 1175 +++++++++++++++++++++++++++++++++++++-
6 files changed, 1416 insertions(+), 59 deletions(-)
create mode 100644 arch/sparc/net/bpf_jit_64.h
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 68ac5c7..d765392 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -31,7 +31,8 @@ config SPARC
select ARCH_WANT_IPC_PARSE_VERSION
select GENERIC_PCI_IOMAP
select HAVE_NMI_WATCHDOG if SPARC64
- select HAVE_CBPF_JIT
+ select HAVE_CBPF_JIT if SPARC32
+ select HAVE_EBPF_JIT if SPARC64
select HAVE_DEBUG_BUGVERBOSE
select GENERIC_SMP_IDLE_THREAD
select GENERIC_CLOCKEVENTS
diff --git a/arch/sparc/net/bpf_jit_64.h b/arch/sparc/net/bpf_jit_64.h
new file mode 100644
index 0000000..331de10
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_64.h
@@ -0,0 +1,79 @@
+#ifndef _BPF_JIT_H
+#define _BPF_JIT_H
+
+/* Conventions:
+ * %g1 : temporary
+ * %g2 : Secondary temporary used by SKB data helper stubs.
+ * %g3 : packet offset passed into SKB data helper stubs.
+ * %o0 : pointer to skb (first argument given to JIT function)
+ * %o1 : BPF A accumulator
+ * %o2 : BPF X accumulator
+ * %o3 : Holds saved %o7 so we can call helper functions without needing
+ * to allocate a register window.
+ * %o4 : skb->len - skb->data_len
+ * %o5 : skb->data
+ */
+
+#ifndef __ASSEMBLER__
+#define G0 0x00
+#define G1 0x01
+#define G2 0x02
+#define G3 0x03
+#define G6 0x06
+#define G7 0x07
+#define O0 0x08
+#define O1 0x09
+#define O2 0x0a
+#define O3 0x0b
+#define O4 0x0c
+#define O5 0x0d
+#define SP 0x0e
+#define O7 0x0f
+#define L0 0x10
+#define L1 0x11
+#define L2 0x12
+#define L3 0x13
+#define L4 0x14
+#define L5 0x15
+#define L6 0x16
+#define L7 0x17
+#define I0 0x18
+#define I1 0x19
+#define I2 0x1a
+#define I3 0x1b
+#define I4 0x1c
+#define I5 0x1d
+#define FP 0x1e
+#define I7 0x1f
+
+#define r_SKB L0
+#define r_HEADLEN L4
+#define r_SKB_DATA L5
+#define r_TMP G1
+#define r_TMP2 G3
+
+/* assembly code in arch/sparc/net/bpf_jit_asm.S */
+extern u32 bpf_jit_load_word[];
+extern u32 bpf_jit_load_half[];
+extern u32 bpf_jit_load_byte[];
+extern u32 bpf_jit_load_byte_msh[];
+extern u32 bpf_jit_load_word_positive_offset[];
+extern u32 bpf_jit_load_half_positive_offset[];
+extern u32 bpf_jit_load_byte_positive_offset[];
+extern u32 bpf_jit_load_byte_msh_positive_offset[];
+extern u32 bpf_jit_load_word_negative_offset[];
+extern u32 bpf_jit_load_half_negative_offset[];
+extern u32 bpf_jit_load_byte_negative_offset[];
+extern u32 bpf_jit_load_byte_msh_negative_offset[];
+
+#else
+#define r_RESULT %o0
+#define r_SKB %o0
+#define r_OFF %o1
+#define r_HEADLEN %l4
+#define r_SKB_DATA %l5
+#define r_TMP %g1
+#define r_TMP2 %g3
+#endif
+
+#endif /* _BPF_JIT_H */
diff --git a/arch/sparc/net/bpf_jit_asm_32.S b/arch/sparc/net/bpf_jit_asm_32.S
index 5632cdc..dcc402f 100644
--- a/arch/sparc/net/bpf_jit_asm_32.S
+++ b/arch/sparc/net/bpf_jit_asm_32.S
@@ -2,17 +2,10 @@
#include "bpf_jit_32.h"
-#ifdef CONFIG_SPARC64
-#define SAVE_SZ 176
-#define SCRATCH_OFF STACK_BIAS + 128
-#define BE_PTR(label) be,pn %xcc, label
-#define SIGN_EXTEND(reg) sra reg, 0, reg
-#else
#define SAVE_SZ 96
#define SCRATCH_OFF 72
#define BE_PTR(label) be label
#define SIGN_EXTEND(reg)
-#endif
#define SKF_MAX_NEG_OFF (-0x200000) /* SKF_LL_OFF from filter.h */
diff --git a/arch/sparc/net/bpf_jit_asm_64.S b/arch/sparc/net/bpf_jit_asm_64.S
index 6fb023f..3b3f146 100644
--- a/arch/sparc/net/bpf_jit_asm_64.S
+++ b/arch/sparc/net/bpf_jit_asm_64.S
@@ -1 +1,161 @@
-#include "bpf_jit_asm_32.S"
+#include <asm/ptrace.h>
+
+#include "bpf_jit_64.h"
+
+#define SAVE_SZ 176
+#define SCRATCH_OFF STACK_BIAS + 128
+#define BE_PTR(label) be,pn %xcc, label
+#define SIGN_EXTEND(reg) sra reg, 0, reg
+
+#define SKF_MAX_NEG_OFF (-0x200000) /* SKF_LL_OFF from filter.h */
+
+ .text
+ .globl bpf_jit_load_word
+bpf_jit_load_word:
+ cmp r_OFF, 0
+ bl bpf_slow_path_word_neg
+ nop
+ .globl bpf_jit_load_word_positive_offset
+bpf_jit_load_word_positive_offset:
+ sub r_HEADLEN, r_OFF, r_TMP
+ cmp r_TMP, 3
+ ble bpf_slow_path_word
+ add r_SKB_DATA, r_OFF, r_TMP
+ andcc r_TMP, 3, %g0
+ bne load_word_unaligned
+ nop
+ retl
+ ld [r_TMP], r_RESULT
+load_word_unaligned:
+ ldub [r_TMP + 0x0], r_OFF
+ ldub [r_TMP + 0x1], r_TMP2
+ sll r_OFF, 8, r_OFF
+ or r_OFF, r_TMP2, r_OFF
+ ldub [r_TMP + 0x2], r_TMP2
+ sll r_OFF, 8, r_OFF
+ or r_OFF, r_TMP2, r_OFF
+ ldub [r_TMP + 0x3], r_TMP2
+ sll r_OFF, 8, r_OFF
+ retl
+ or r_OFF, r_TMP2, r_RESULT
+
+ .globl bpf_jit_load_half
+bpf_jit_load_half:
+ cmp r_OFF, 0
+ bl bpf_slow_path_half_neg
+ nop
+ .globl bpf_jit_load_half_positive_offset
+bpf_jit_load_half_positive_offset:
+ sub r_HEADLEN, r_OFF, r_TMP
+ cmp r_TMP, 1
+ ble bpf_slow_path_half
+ add r_SKB_DATA, r_OFF, r_TMP
+ andcc r_TMP, 1, %g0
+ bne load_half_unaligned
+ nop
+ retl
+ lduh [r_TMP], r_RESULT
+load_half_unaligned:
+ ldub [r_TMP + 0x0], r_OFF
+ ldub [r_TMP + 0x1], r_TMP2
+ sll r_OFF, 8, r_OFF
+ retl
+ or r_OFF, r_TMP2, r_RESULT
+
+ .globl bpf_jit_load_byte
+bpf_jit_load_byte:
+ cmp r_OFF, 0
+ bl bpf_slow_path_byte_neg
+ nop
+ .globl bpf_jit_load_byte_positive_offset
+bpf_jit_load_byte_positive_offset:
+ cmp r_OFF, r_HEADLEN
+ bge bpf_slow_path_byte
+ nop
+ retl
+ ldub [r_SKB_DATA + r_OFF], r_RESULT
+
+#define bpf_slow_path_common(LEN) \
+ save %sp, -SAVE_SZ, %sp; \
+ mov %i0, %o0; \
+ mov %i1, %o1; \
+ add %fp, SCRATCH_OFF, %o2; \
+ call skb_copy_bits; \
+ mov (LEN), %o3; \
+ cmp %o0, 0; \
+ restore;
+
+bpf_slow_path_word:
+ bpf_slow_path_common(4)
+ bl bpf_error
+ ld [%sp + SCRATCH_OFF], r_RESULT
+ retl
+ nop
+bpf_slow_path_half:
+ bpf_slow_path_common(2)
+ bl bpf_error
+ lduh [%sp + SCRATCH_OFF], r_RESULT
+ retl
+ nop
+bpf_slow_path_byte:
+ bpf_slow_path_common(1)
+ bl bpf_error
+ ldub [%sp + SCRATCH_OFF], r_RESULT
+ retl
+ nop
+
+#define bpf_negative_common(LEN) \
+ save %sp, -SAVE_SZ, %sp; \
+ mov %i0, %o0; \
+ mov %i1, %o1; \
+ SIGN_EXTEND(%o1); \
+ call bpf_internal_load_pointer_neg_helper; \
+ mov (LEN), %o2; \
+ mov %o0, r_TMP; \
+ cmp %o0, 0; \
+ BE_PTR(bpf_error); \
+ restore;
+
+bpf_slow_path_word_neg:
+ sethi %hi(SKF_MAX_NEG_OFF), r_TMP
+ cmp r_OFF, r_TMP
+ bl bpf_error
+ nop
+ .globl bpf_jit_load_word_negative_offset
+bpf_jit_load_word_negative_offset:
+ bpf_negative_common(4)
+ andcc r_TMP, 3, %g0
+ bne load_word_unaligned
+ nop
+ retl
+ ld [r_TMP], r_RESULT
+
+bpf_slow_path_half_neg:
+ sethi %hi(SKF_MAX_NEG_OFF), r_TMP
+ cmp r_OFF, r_TMP
+ bl bpf_error
+ nop
+ .globl bpf_jit_load_half_negative_offset
+bpf_jit_load_half_negative_offset:
+ bpf_negative_common(2)
+ andcc r_TMP, 1, %g0
+ bne load_half_unaligned
+ nop
+ retl
+ lduh [r_TMP], r_RESULT
+
+bpf_slow_path_byte_neg:
+ sethi %hi(SKF_MAX_NEG_OFF), r_TMP
+ cmp r_OFF, r_TMP
+ bl bpf_error
+ nop
+ .globl bpf_jit_load_byte_negative_offset
+bpf_jit_load_byte_negative_offset:
+ bpf_negative_common(1)
+ retl
+ ldub [r_TMP], r_RESULT
+
+bpf_error:
+ /* Make the JIT program itself return zero. */
+ ret
+ restore %g0, %g0, %o0
diff --git a/arch/sparc/net/bpf_jit_comp_32.c b/arch/sparc/net/bpf_jit_comp_32.c
index 83fc41d..d193748 100644
--- a/arch/sparc/net/bpf_jit_comp_32.c
+++ b/arch/sparc/net/bpf_jit_comp_32.c
@@ -17,24 +17,6 @@ static inline bool is_simm13(unsigned int value)
return value + 0x1000 < 0x2000;
}
-static void bpf_flush_icache(void *start_, void *end_)
-{
-#ifdef CONFIG_SPARC64
- /* Cheetah's I-cache is fully coherent. */
- if (tlb_type == spitfire) {
- unsigned long start = (unsigned long) start_;
- unsigned long end = (unsigned long) end_;
-
- start &= ~7UL;
- end = (end + 7UL) & ~7UL;
- while (start < end) {
- flushi(start);
- start += 32;
- }
- }
-#endif
-}
-
#define SEEN_DATAREF 1 /* might call external helpers */
#define SEEN_XREG 2 /* ebx is used */
#define SEEN_MEM 4 /* use mem[] for temporary storage */
@@ -82,11 +64,7 @@ static void bpf_flush_icache(void *start_, void *end_)
#define BE (F2(0, 2) | CONDE)
#define BNE (F2(0, 2) | CONDNE)
-#ifdef CONFIG_SPARC64
-#define BE_PTR (F2(0, 1) | CONDE | (2 << 20))
-#else
#define BE_PTR BE
-#endif
#define SETHI(K, REG) \
(F2(0, 0x4) | RD(REG) | (((K) >> 10) & 0x3fffff))
@@ -116,13 +94,8 @@ static void bpf_flush_icache(void *start_, void *end_)
#define LD64 F3(3, 0x0b)
#define ST32 F3(3, 0x04)
-#ifdef CONFIG_SPARC64
-#define LDPTR LD64
-#define BASE_STACKFRAME 176
-#else
#define LDPTR LD32
#define BASE_STACKFRAME 96
-#endif
#define LD32I (LD32 | IMMED)
#define LD8I (LD8 | IMMED)
@@ -234,11 +207,7 @@ do { BUILD_BUG_ON(FIELD_SIZEOF(STRUCT, FIELD) != sizeof(u8)); \
__emit_load8(BASE, STRUCT, FIELD, DEST); \
} while (0)
-#ifdef CONFIG_SPARC64
-#define BIAS (STACK_BIAS - 4)
-#else
#define BIAS (-4)
-#endif
#define emit_ldmem(OFF, DEST) \
do { *prog++ = LD32I | RS1(SP) | S13(BIAS - (OFF)) | RD(DEST); \
@@ -249,13 +218,8 @@ do { *prog++ = ST32I | RS1(SP) | S13(BIAS - (OFF)) | RD(SRC); \
} while (0)
#ifdef CONFIG_SMP
-#ifdef CONFIG_SPARC64
-#define emit_load_cpu(REG) \
- emit_load16(G6, struct thread_info, cpu, REG)
-#else
#define emit_load_cpu(REG) \
emit_load32(G6, struct thread_info, cpu, REG)
-#endif
#else
#define emit_load_cpu(REG) emit_clear(REG)
#endif
@@ -486,7 +450,6 @@ void bpf_jit_compile(struct bpf_prog *fp)
if (K == 1)
break;
emit_write_y(G0);
-#ifdef CONFIG_SPARC32
/* The Sparc v8 architecture requires
* three instructions between a %y
* register write and the first use.
@@ -494,31 +457,21 @@ void bpf_jit_compile(struct bpf_prog *fp)
emit_nop();
emit_nop();
emit_nop();
-#endif
emit_alu_K(DIV, K);
break;
case BPF_ALU | BPF_DIV | BPF_X: /* A /= X; */
emit_cmpi(r_X, 0);
if (pc_ret0 > 0) {
t_offset = addrs[pc_ret0 - 1];
-#ifdef CONFIG_SPARC32
emit_branch(BE, t_offset + 20);
-#else
- emit_branch(BE, t_offset + 8);
-#endif
emit_nop(); /* delay slot */
} else {
emit_branch_off(BNE, 16);
emit_nop();
-#ifdef CONFIG_SPARC32
emit_jump(cleanup_addr + 20);
-#else
- emit_jump(cleanup_addr + 8);
-#endif
emit_clear(r_A);
}
emit_write_y(G0);
-#ifdef CONFIG_SPARC32
/* The Sparc v8 architecture requires
* three instructions between a %y
* register write and the first use.
@@ -526,7 +479,6 @@ void bpf_jit_compile(struct bpf_prog *fp)
emit_nop();
emit_nop();
emit_nop();
-#endif
emit_alu_X(DIV);
break;
case BPF_ALU | BPF_NEG:
@@ -797,7 +749,6 @@ cond_branch: f_offset = addrs[i + filter[i].jf];
bpf_jit_dump(flen, proglen, pass + 1, image);
if (image) {
- bpf_flush_icache(image, image + proglen);
fp->bpf_func = (void *)image;
fp->jited = 1;
}
diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c
index 49b5f65..e16315f 100644
--- a/arch/sparc/net/bpf_jit_comp_64.c
+++ b/arch/sparc/net/bpf_jit_comp_64.c
@@ -1 +1,1174 @@
-#include "bpf_jit_comp_32.c"
+#include <linux/moduleloader.h>
+#include <linux/workqueue.h>
+#include <linux/netdevice.h>
+#include <linux/filter.h>
+#include <linux/cache.h>
+#include <linux/if_vlan.h>
+
+#include <asm/cacheflush.h>
+#include <asm/ptrace.h>
+
+#include "bpf_jit_64.h"
+
+int bpf_jit_enable __read_mostly;
+
+static inline bool is_simm13(unsigned int value)
+{
+ return value + 0x1000 < 0x2000;
+}
+
+static void bpf_flush_icache(void *start_, void *end_)
+{
+ /* Cheetah's I-cache is fully coherent. */
+ if (tlb_type == spitfire) {
+ unsigned long start = (unsigned long) start_;
+ unsigned long end = (unsigned long) end_;
+
+ start &= ~7UL;
+ end = (end + 7UL) & ~7UL;
+ while (start < end) {
+ flushi(start);
+ start += 32;
+ }
+ }
+}
+
+#define SEEN_DATAREF 1 /* might call external helpers */
+#define SEEN_XREG 2 /* ebx is used */
+#define SEEN_MEM 4 /* use mem[] for temporary storage */
+
+#define S13(X) ((X) & 0x1fff)
+#define IMMED 0x00002000
+#define RD(X) ((X) << 25)
+#define RS1(X) ((X) << 14)
+#define RS2(X) ((X))
+#define OP(X) ((X) << 30)
+#define OP2(X) ((X) << 22)
+#define OP3(X) ((X) << 19)
+#define COND(X) ((X) << 25)
+#define F1(X) OP(X)
+#define F2(X, Y) (OP(X) | OP2(Y))
+#define F3(X, Y) (OP(X) | OP3(Y))
+#define ASI(X) (((X) & 0xff) << 5)
+
+#define CONDN COND(0x0)
+#define CONDE COND(0x1)
+#define CONDLE COND(0x2)
+#define CONDL COND(0x3)
+#define CONDLEU COND(0x4)
+#define CONDCS COND(0x5)
+#define CONDNEG COND(0x6)
+#define CONDVC COND(0x7)
+#define CONDA COND(0x8)
+#define CONDNE COND(0x9)
+#define CONDG COND(0xa)
+#define CONDGE COND(0xb)
+#define CONDGU COND(0xc)
+#define CONDCC COND(0xd)
+#define CONDPOS COND(0xe)
+#define CONDVS COND(0xf)
+
+#define CONDGEU CONDCC
+#define CONDLU CONDCS
+
+#define WDISP22(X) (((X) >> 2) & 0x3fffff)
+#define WDISP19(X) (((X) >> 2) & 0x7ffff)
+
+#define ANNUL (1 << 29)
+#define XCC (1 << 21)
+
+#define BRANCH (F2(0, 1) | XCC)
+
+#define BA (BRANCH | CONDA)
+#define BG (BRANCH | CONDG)
+#define BGU (BRANCH | CONDGU)
+#define BLEU (BRANCH | CONDLEU)
+#define BGE (BRANCH | CONDGE)
+#define BGEU (BRANCH | CONDGEU)
+#define BLU (BRANCH | CONDLU)
+#define BE (BRANCH | CONDE)
+#define BNE (BRANCH | CONDNE)
+
+#define SETHI(K, REG) \
+ (F2(0, 0x4) | RD(REG) | (((K) >> 10) & 0x3fffff))
+#define OR_LO(K, REG) \
+ (F3(2, 0x02) | IMMED | RS1(REG) | ((K) & 0x3ff) | RD(REG))
+
+#define ADD F3(2, 0x00)
+#define AND F3(2, 0x01)
+#define ANDCC F3(2, 0x11)
+#define OR F3(2, 0x02)
+#define XOR F3(2, 0x03)
+#define SUB F3(2, 0x04)
+#define SUBCC F3(2, 0x14)
+#define MUL F3(2, 0x0a)
+#define MULX F3(2, 0x09)
+#define UDIVX F3(2, 0x0d)
+#define DIV F3(2, 0x0e)
+#define SLL F3(2, 0x25)
+#define SLLX (F3(2, 0x25)|(1<<12))
+#define SRA F3(2, 0x27)
+#define SRAX (F3(2, 0x27)|(1<<12))
+#define SRL F3(2, 0x26)
+#define SRLX (F3(2, 0x26)|(1<<12))
+#define JMPL F3(2, 0x38)
+#define SAVE F3(2, 0x3c)
+#define RESTORE F3(2, 0x3d)
+#define CALL F1(1)
+#define BR F2(0, 0x01)
+#define RD_Y F3(2, 0x28)
+#define WR_Y F3(2, 0x30)
+
+#define LD32 F3(3, 0x00)
+#define LD8 F3(3, 0x01)
+#define LD16 F3(3, 0x02)
+#define LD64 F3(3, 0x0b)
+#define LD64A F3(3, 0x1b)
+#define ST8 F3(3, 0x05)
+#define ST16 F3(3, 0x06)
+#define ST32 F3(3, 0x04)
+#define ST64 F3(3, 0x0e)
+
+#define CAS F3(3, 0x3c)
+#define CASX F3(3, 0x3e)
+
+#define LDPTR LD64
+#define BASE_STACKFRAME 176
+
+#define LD32I (LD32 | IMMED)
+#define LD8I (LD8 | IMMED)
+#define LD16I (LD16 | IMMED)
+#define LD64I (LD64 | IMMED)
+#define LDPTRI (LDPTR | IMMED)
+#define ST32I (ST32 | IMMED)
+
+struct jit_ctx {
+ struct bpf_prog *prog;
+ unsigned int *offset;
+ int idx;
+ int epilogue_offset;
+ bool tmp_1_used;
+ bool tmp_2_used;
+ bool tmp_3_used;
+ bool saw_ld_abs_ind;
+ bool saw_frame_pointer;
+ u32 *image;
+};
+
+#define TMP_REG_1 (MAX_BPF_JIT_REG + 0)
+#define TMP_REG_2 (MAX_BPF_JIT_REG + 1)
+#define SKB_HLEN_REG (MAX_BPF_JIT_REG + 2)
+#define SKB_DATA_REG (MAX_BPF_JIT_REG + 3)
+#define TMP_REG_3 (MAX_BPF_JIT_REG + 4)
+
+/* Map BPF registers to SPARC registers */
+static const int bpf2sparc[] = {
+ /* return value from in-kernel function, and exit value from eBPF */
+ [BPF_REG_0] = O5,
+
+ /* arguments from eBPF program to in-kernel function */
+ [BPF_REG_1] = O0,
+ [BPF_REG_2] = O1,
+ [BPF_REG_3] = O2,
+ [BPF_REG_4] = O3,
+ [BPF_REG_5] = O4,
+
+ /* callee saved registers that in-kernel function will preserve */
+ [BPF_REG_6] = L0,
+ [BPF_REG_7] = L1,
+ [BPF_REG_8] = L2,
+ [BPF_REG_9] = L3,
+
+ /* read-only frame pointer to access stack */
+ [BPF_REG_FP] = FP,
+
+ [BPF_REG_AX] = G7,
+
+ /* temporary register for internal BPF JIT */
+ [TMP_REG_1] = G1,
+ [TMP_REG_2] = G2,
+ [TMP_REG_3] = G3,
+
+ [SKB_HLEN_REG] = L4,
+ [SKB_DATA_REG] = L5,
+};
+
+static void emit(const u32 insn, struct jit_ctx *ctx)
+{
+ if (ctx->image != NULL)
+ ctx->image[ctx->idx] = insn;
+
+ ctx->idx++;
+}
+
+static void emit_call(u32 *func, struct jit_ctx *ctx)
+{
+ if (ctx->image != NULL) {
+ void *here = &ctx->image[ctx->idx];
+ unsigned int off;
+
+ off = (void *)func - here;
+ ctx->image[ctx->idx] = CALL | ((off >> 2) & 0x3fffffff);
+ }
+ ctx->idx++;
+}
+
+static void emit_nop(struct jit_ctx *ctx)
+{
+ emit(SETHI(0, G0), ctx);
+}
+
+static void emit_reg_move(u32 from, u32 to, struct jit_ctx *ctx)
+{
+ emit(OR | RS1(G0) | RS2(from) | RD(to), ctx);
+}
+
+/* Emit 32-bit constant, zero extended. */
+static void emit_set_const(s32 K, u32 reg, struct jit_ctx *ctx)
+{
+ emit(SETHI(K, reg), ctx);
+ emit(OR_LO(K, reg), ctx);
+}
+
+/* Emit 32-bit constant, sign extended. */
+static void emit_set_const_sext(s32 K, u32 reg, struct jit_ctx *ctx)
+{
+ if (K >= 0) {
+ emit(SETHI(K, reg), ctx);
+ emit(OR_LO(K, reg), ctx);
+ } else {
+ u32 hbits = ~(u32) K;
+ u32 lbits = -0x400 | (u32) K;
+
+ emit(SETHI(hbits, reg), ctx);
+ emit(XOR | IMMED | RS1(reg) | S13(lbits) | RD(reg), ctx);
+ }
+}
+
+/* Emit: OP DST, SRC, DST */
+static void emit_alu(u32 opcode, u32 src, u32 dst, struct jit_ctx *ctx)
+{
+ emit(opcode | RS1(dst) | RS2(src) | RD(dst), ctx);
+}
+
+/* Emit: OP A, B, C */
+static void emit_alu3(u32 opcode, u32 a, u32 b, u32 c, struct jit_ctx *ctx)
+{
+ emit(opcode | RS1(a) | RS2(b) | RD(c), ctx);
+}
+
+/* Emit either:
+ *
+ * OP DST, K, DST
+ *
+ * or
+ *
+ * sethi %hi(K), r_TMP
+ * or r_TMP, %lo(K), r_TMP
+ * OP DST, r_TMP, DST
+ *
+ * depending upon whether K fits in a signed 13-bit
+ * immediate instruction field.
+ */
+static void emit_alu_K(unsigned int opcode, unsigned int dst, unsigned int imm,
+ struct jit_ctx *ctx)
+{
+ bool small_immed = is_simm13(imm);
+ unsigned int insn = opcode;
+
+ insn |= RS1(dst) | RD(dst);
+ if (small_immed) {
+ emit(insn | IMMED | S13(imm), ctx);
+ } else {
+ unsigned int tmp = bpf2sparc[TMP_REG_1];
+
+ ctx->tmp_1_used = true;
+
+ emit_set_const_sext(imm, tmp, ctx);
+ emit(insn | RS2(tmp), ctx);
+ }
+}
+
+/* Emit either:
+ *
+ * OP SRC, K, DST
+ *
+ * or
+ *
+ * sethi %hi(K), r_TMP
+ * or r_TMP, %lo(K), r_TMP
+ * OP SRC, r_TMP, DST
+ *
+ * depending upon whether K fits in a signed 13-bit
+ * immediate instruction field.
+ */
+static void emit_alu3_K(unsigned int opcode, unsigned int src, unsigned int imm,
+ unsigned int dst, struct jit_ctx *ctx)
+{
+ bool small_immed = is_simm13(imm);
+ unsigned int insn = opcode;
+
+ insn |= RS1(src) | RD(dst);
+ if (small_immed) {
+ emit(insn | IMMED | S13(imm), ctx);
+ } else {
+ unsigned int tmp = bpf2sparc[TMP_REG_1];
+
+ ctx->tmp_1_used = true;
+
+ emit_set_const_sext(imm, tmp, ctx);
+ emit(insn | RS2(tmp), ctx);
+ }
+}
+
+static void emit_loadimm32(s32 K, unsigned int dest, struct jit_ctx *ctx)
+{
+ if (K >= 0 && is_simm13(K)) {
+ /* or %g0, K, DEST */
+ emit(OR | IMMED | RS1(G0) | S13(K) | RD(dest), ctx);
+ } else {
+ emit_set_const(K, dest, ctx);
+ }
+}
+
+static void emit_loadimm(s32 K, unsigned int dest, struct jit_ctx *ctx)
+{
+ if (is_simm13(K)) {
+ /* or %g0, K, DEST */
+ emit(OR | IMMED | RS1(G0) | S13(K) | RD(dest), ctx);
+ } else {
+ emit_set_const(K, dest, ctx);
+ }
+}
+
+static void emit_loadimm_sext(s32 K, unsigned int dest, struct jit_ctx *ctx)
+{
+ if (is_simm13(K)) {
+ /* or %g0, K, DEST */
+ emit(OR | IMMED | RS1(G0) | S13(K) | RD(dest), ctx);
+ } else {
+ emit_set_const_sext(K, dest, ctx);
+ }
+}
+
+static void emit_loadimm64(u64 K, unsigned int dest, struct jit_ctx *ctx)
+{
+ unsigned int tmp = bpf2sparc[TMP_REG_1];
+ u32 high_part = (K >> 32);
+ u32 low_part = (K & 0xffffffff);
+
+ ctx->tmp_1_used = true;
+
+ emit_set_const(high_part, tmp, ctx);
+ emit_set_const(low_part, dest, ctx);
+ emit_alu_K(SLLX, tmp, 32, ctx);
+ emit(OR | RS1(dest) | RS2(tmp) | RD(dest), ctx);
+}
+
+static void emit_branch(unsigned int br_opc, unsigned int from_idx, unsigned int to_idx,
+ struct jit_ctx *ctx)
+{
+ unsigned int off = to_idx - from_idx;
+
+ if (br_opc & XCC)
+ emit(br_opc | WDISP19(off << 2), ctx);
+ else
+ emit(br_opc | WDISP22(off << 2), ctx);
+}
+
+#define emit_read_y(REG, CTX) emit(RD_Y | RD(REG), CTX)
+#define emit_write_y(REG, CTX) emit(WR_Y | IMMED | RS1(REG) | S13(0), CTX)
+
+#define emit_cmp(R1, R2, CTX) \
+ emit(SUBCC | RS1(R1) | RS2(R2) | RD(G0), CTX)
+
+#define emit_cmpi(R1, IMM, CTX) \
+ emit(SUBCC | IMMED | RS1(R1) | S13(IMM) | RD(G0), CTX);
+
+#define emit_btst(R1, R2, CTX) \
+ emit(ANDCC | RS1(R1) | RS2(R2) | RD(G0), CTX)
+
+#define emit_btsti(R1, IMM, CTX) \
+ emit(ANDCC | IMMED | RS1(R1) | S13(IMM) | RD(G0), CTX)
+
+static void load_skb_regs(struct jit_ctx *ctx, u8 r_skb)
+{
+ const u8 r_headlen = bpf2sparc[SKB_HLEN_REG];
+ const u8 r_data = bpf2sparc[SKB_DATA_REG];
+ const u8 r_tmp = bpf2sparc[TMP_REG_1];
+ unsigned int off;
+
+ off = offsetof(struct sk_buff, len);
+ emit(LD32I | RS1(r_skb) | S13(off) | RD(r_headlen), ctx);
+
+ off = offsetof(struct sk_buff, data_len);
+ emit(LD32I | RS1(r_skb) | S13(off) | RD(r_tmp), ctx);
+
+ emit(SUB | RS1(r_headlen) | RS2(r_tmp) | RD(r_headlen), ctx);
+
+ off = offsetof(struct sk_buff, data);
+ emit(LDPTRI | RS1(r_skb) | S13(off) | RD(r_data), ctx);
+}
+
+static void build_prologue(struct jit_ctx *ctx)
+{
+ s32 stack_needed = BASE_STACKFRAME;
+
+ if (ctx->saw_frame_pointer)
+ stack_needed += MAX_BPF_STACK;
+
+ /* save %sp, -176, %sp */
+ emit(SAVE | IMMED | RS1(SP) | S13(-stack_needed) | RD(SP), ctx);
+
+ emit_reg_move(I0, O0, ctx);
+
+ if (ctx->saw_ld_abs_ind)
+ load_skb_regs(ctx, bpf2sparc[BPF_REG_1]);
+}
+
+static void build_epilogue(struct jit_ctx *ctx)
+{
+ ctx->epilogue_offset = ctx->idx;
+
+ /* ret (jmpl %i7 + 8, %g0) */
+ emit(JMPL | IMMED | RS1(I7) | S13(8) | RD(G0), ctx);
+
+ /* restore %i5, %g0, %o0 */
+ emit(RESTORE | RS1(bpf2sparc[BPF_REG_0]) | RS2(G0) | RD(O0), ctx);
+}
+
+static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx)
+{
+ const u8 code = insn->code;
+ const u8 dst = bpf2sparc[insn->dst_reg];
+ const u8 src = bpf2sparc[insn->src_reg];
+ const int i = insn - ctx->prog->insnsi;
+ const s16 off = insn->off;
+ const s32 imm = insn->imm;
+ u32 *func;
+
+ /* The verifier makes sure we don't write to the FP, so we
+ * need not check that here.
+ */
+ if (insn->src_reg == BPF_REG_FP) {
+ ctx->saw_frame_pointer = true;
+ if (BPF_CLASS(code) == BPF_ALU ||
+ (BPF_CLASS(code) == BPF_ALU64 && BPF_OP(code) != BPF_MOV)) {
+ pr_err_once("JIT: non-64bit-MOV ALU on FP\n");
+ return -EINVAL;
+ }
+ }
+
+ switch (code) {
+ /* dst = src */
+ case BPF_ALU | BPF_MOV | BPF_X:
+ emit_alu3_K(SRL, src, 0, dst, ctx);
+ break;
+ case BPF_ALU64 | BPF_MOV | BPF_X:
+ if (src == FP) {
+ /* Apply stack bias */
+ emit_alu3_K(ADD, src, STACK_BIAS, dst, ctx);
+ } else {
+ emit_reg_move(src, dst, ctx);
+ }
+ break;
+ /* dst = dst OP src */
+ case BPF_ALU | BPF_ADD | BPF_X:
+ case BPF_ALU64 | BPF_ADD | BPF_X:
+ emit_alu(ADD, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_SUB | BPF_X:
+ case BPF_ALU64 | BPF_SUB | BPF_X:
+ emit_alu(SUB, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_AND | BPF_X:
+ case BPF_ALU64 | BPF_AND | BPF_X:
+ emit_alu(AND, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_OR | BPF_X:
+ case BPF_ALU64 | BPF_OR | BPF_X:
+ emit_alu(OR, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_XOR | BPF_X:
+ case BPF_ALU64 | BPF_XOR | BPF_X:
+ emit_alu(XOR, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_MUL | BPF_X:
+ emit_alu(MUL, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_MUL | BPF_X:
+ emit_alu(MULX, src, dst, ctx);
+ break;
+ case BPF_ALU | BPF_DIV | BPF_X:
+ emit_cmp(src, G0, ctx);
+ emit_branch(BE|ANNUL, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_loadimm(0, bpf2sparc[BPF_REG_0], ctx);
+
+ emit_write_y(G0, ctx);
+ emit_alu(DIV, src, dst, ctx);
+ break;
+
+ case BPF_ALU64 | BPF_DIV | BPF_X:
+ emit_cmp(src, G0, ctx);
+ emit_branch(BE|ANNUL, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_loadimm(0, bpf2sparc[BPF_REG_0], ctx);
+
+ emit_alu(UDIVX, src, dst, ctx);
+ break;
+
+ case BPF_ALU | BPF_MOD | BPF_X: {
+ unsigned int tmp = bpf2sparc[TMP_REG_1];
+
+ ctx->tmp_1_used = true;
+
+ emit_cmp(src, G0, ctx);
+ emit_branch(BE|ANNUL, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_loadimm(0, bpf2sparc[BPF_REG_0], ctx);
+
+ emit_write_y(G0, ctx);
+ emit_alu3(DIV, dst, src, tmp, ctx);
+ emit_alu3(MULX, tmp, src, tmp, ctx);
+ emit_alu3(SUB, dst, tmp, dst, ctx);
+ goto do_alu32_trunc;
+ }
+ case BPF_ALU64 | BPF_MOD | BPF_X: {
+ unsigned int tmp = bpf2sparc[TMP_REG_1];
+
+ ctx->tmp_1_used = true;
+
+ emit_cmp(src, G0, ctx);
+ emit_branch(BE|ANNUL, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_loadimm(0, bpf2sparc[BPF_REG_0], ctx);
+
+ emit_alu3(UDIVX, dst, src, tmp, ctx);
+ emit_alu3(MULX, tmp, src, tmp, ctx);
+ emit_alu3(SUB, dst, tmp, dst, ctx);
+ break;
+ }
+ case BPF_ALU | BPF_LSH | BPF_X:
+ emit_alu(SLL, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_LSH | BPF_X:
+ emit_alu(SLLX, src, dst, ctx);
+ break;
+ case BPF_ALU | BPF_RSH | BPF_X:
+ emit_alu(SRL, src, dst, ctx);
+ break;
+ case BPF_ALU64 | BPF_RSH | BPF_X:
+ emit_alu(SRLX, src, dst, ctx);
+ break;
+ case BPF_ALU | BPF_ARSH | BPF_X:
+ emit_alu(SRA, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_ARSH | BPF_X:
+ emit_alu(SRAX, src, dst, ctx);
+ break;
+
+ /* dst = -dst */
+ case BPF_ALU | BPF_NEG:
+ case BPF_ALU64 | BPF_NEG:
+ emit(SUB | RS1(0) | RS2(dst) | RD(dst), ctx);
+ goto do_alu32_trunc;
+
+ case BPF_ALU | BPF_END | BPF_FROM_BE:
+ switch (imm) {
+ case 16:
+ emit_alu_K(SLL, dst, 16, ctx);
+ emit_alu_K(SRL, dst, 16, ctx);
+ break;
+ case 32:
+ emit_alu_K(SRL, dst, 0, ctx);
+ break;
+ case 64:
+ /* nop */
+ break;
+
+ }
+ break;
+
+ /* dst = BSWAP##imm(dst) */
+ case BPF_ALU | BPF_END | BPF_FROM_LE: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ const u8 tmp2 = bpf2sparc[TMP_REG_2];
+
+ ctx->tmp_1_used = true;
+ switch (imm) {
+ case 16:
+ emit_alu3_K(AND, dst, 0xff, tmp, ctx);
+ emit_alu3_K(SRL, dst, 8, dst, ctx);
+ emit_alu3_K(AND, dst, 0xff, dst, ctx);
+ emit_alu3_K(SLL, tmp, 8, tmp, ctx);
+ emit_alu(OR, tmp, dst, ctx);
+ break;
+
+ case 32:
+ ctx->tmp_2_used = true;
+ emit_alu3_K(SRL, dst, 24, tmp, ctx); /* tmp = dst >> 24 */
+ emit_alu3_K(SRL, dst, 16, tmp2, ctx); /* tmp2 = dst >> 16 */
+ emit_alu3_K(AND, tmp2, 0xff, tmp2, ctx);/* tmp2 = tmp2 & 0xff */
+ emit_alu3_K(SLL, tmp2, 8, tmp2, ctx); /* tmp2 = tmp2 << 8 */
+ emit_alu(OR, tmp2, tmp, ctx); /* tmp = tmp | tmp2 */
+ emit_alu3_K(SRL, dst, 8, tmp2, ctx); /* tmp2 = dst >> 8 */
+ emit_alu3_K(AND, tmp2, 0xff, tmp2, ctx);/* tmp2 = tmp2 & 0xff */
+ emit_alu3_K(SLL, tmp2, 16, tmp2, ctx); /* tmp2 = tmp2 << 16 */
+ emit_alu(OR, tmp2, tmp, ctx); /* tmp = tmp | tmp2 */
+ emit_alu3_K(AND, dst, 0xff, dst, ctx); /* dst = dst & 0xff */
+ emit_alu3_K(SLL, dst, 24, dst, ctx); /* dst = dst << 24 */
+ emit_alu(OR, tmp, dst, ctx); /* dst = dst | tmp */
+ break;
+
+ case 64:
+ emit_alu3_K(ADD, SP, STACK_BIAS + 128, tmp, ctx);
+ emit(ST64 | RS1(tmp) | RS2(G0) | RD(dst), ctx);
+ emit(LD64A | ASI(ASI_PL) | RS1(tmp) | RS2(G0) | RD(dst), ctx);
+ break;
+ }
+ break;
+ }
+ /* dst = imm */
+ case BPF_ALU | BPF_MOV | BPF_K:
+ emit_loadimm32(imm, dst, ctx);
+ break;
+ case BPF_ALU64 | BPF_MOV | BPF_K:
+ emit_loadimm(imm, dst, ctx);
+ break;
+ /* dst = dst OP imm */
+ case BPF_ALU | BPF_ADD | BPF_K:
+ case BPF_ALU64 | BPF_ADD | BPF_K:
+ emit_alu_K(ADD, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_SUB | BPF_K:
+ case BPF_ALU64 | BPF_SUB | BPF_K:
+ emit_alu_K(SUB, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_AND | BPF_K:
+ case BPF_ALU64 | BPF_AND | BPF_K:
+ emit_alu_K(AND, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_OR | BPF_K:
+ case BPF_ALU64 | BPF_OR | BPF_K:
+ emit_alu_K(OR, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_XOR | BPF_K:
+ case BPF_ALU64 | BPF_XOR | BPF_K:
+ emit_alu_K(XOR, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_MUL | BPF_K:
+ emit_alu_K(MUL, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_MUL | BPF_K:
+ emit_alu_K(MULX, dst, imm, ctx);
+ break;
+ case BPF_ALU | BPF_DIV | BPF_K:
+ if (imm == 0)
+ return -EINVAL;
+
+ emit_write_y(G0, ctx);
+ emit_alu_K(DIV, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_DIV | BPF_K:
+ if (imm == 0)
+ return -EINVAL;
+
+ emit_alu_K(UDIVX, dst, imm, ctx);
+ break;
+ case BPF_ALU | BPF_MOD | BPF_K: {
+ unsigned int tmp = bpf2sparc[TMP_REG_2];
+
+ if (imm == 0)
+ return -EINVAL;
+
+ /* XXX Emit non-simm13 constants only once... */
+ ctx->tmp_2_used = true;
+
+ emit_write_y(G0, ctx);
+ emit_alu3_K(DIV, dst, imm, tmp, ctx);
+ emit_alu3_K(MULX, tmp, imm, tmp, ctx);
+ emit_alu3(SUB, dst, tmp, dst, ctx);
+ goto do_alu32_trunc;
+ }
+ case BPF_ALU64 | BPF_MOD | BPF_K: {
+ unsigned int tmp = bpf2sparc[TMP_REG_2];
+
+ if (imm == 0)
+ return -EINVAL;
+
+ /* XXX Emit non-simm13 constants only once... */
+ ctx->tmp_2_used = true;
+
+ emit_alu3_K(UDIVX, dst, imm, tmp, ctx);
+ emit_alu3_K(MULX, tmp, imm, tmp, ctx);
+ emit_alu3(SUB, dst, tmp, dst, ctx);
+ break;
+ }
+ case BPF_ALU | BPF_LSH | BPF_K:
+ emit_alu_K(SLL, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_LSH | BPF_K:
+ emit_alu_K(SLLX, dst, imm, ctx);
+ break;
+ case BPF_ALU | BPF_RSH | BPF_K:
+ emit_alu_K(SRL, dst, imm, ctx);
+ break;
+ case BPF_ALU64 | BPF_RSH | BPF_K:
+ emit_alu_K(SRLX, dst, imm, ctx);
+ break;
+ case BPF_ALU | BPF_ARSH | BPF_K:
+ emit_alu_K(SRA, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_ARSH | BPF_K:
+ emit_alu_K(SRAX, dst, imm, ctx);
+ break;
+
+ do_alu32_trunc:
+ if (BPF_CLASS(code) == BPF_ALU)
+ emit_alu_K(SRL, dst, 0, ctx);
+ break;
+
+ /* JUMP off */
+ case BPF_JMP | BPF_JA:
+ emit_branch(BA, ctx->idx, ctx->offset[i + off], ctx);
+ emit_nop(ctx);
+ break;
+ /* IF (dst COND src) JUMP off */
+ case BPF_JMP | BPF_JEQ | BPF_X:
+ case BPF_JMP | BPF_JGT | BPF_X:
+ case BPF_JMP | BPF_JGE | BPF_X:
+ case BPF_JMP | BPF_JNE | BPF_X:
+ case BPF_JMP | BPF_JSGT | BPF_X:
+ case BPF_JMP | BPF_JSGE | BPF_X: {
+ u32 br_opcode;
+
+ emit_cmp(dst, src, ctx);
+emit_cond_jmp:
+ switch (BPF_OP(code)) {
+ case BPF_JEQ:
+ br_opcode = BE;
+ break;
+ case BPF_JGT:
+ br_opcode = BGU;
+ break;
+ case BPF_JGE:
+ br_opcode = BGEU;
+ break;
+ case BPF_JSET:
+ case BPF_JNE:
+ br_opcode = BNE;
+ break;
+ case BPF_JSGT:
+ br_opcode = BG;
+ break;
+ case BPF_JSGE:
+ br_opcode = BGE;
+ break;
+ default:
+ /* Make sure we dont leak kernel information to the
+ * user.
+ */
+ return -EFAULT;
+ }
+ emit_branch(br_opcode, ctx->idx, ctx->offset[i + off], ctx);
+ emit_nop(ctx);
+ break;
+ }
+ case BPF_JMP | BPF_JSET | BPF_X:
+ emit_btst(dst, src, ctx);
+ goto emit_cond_jmp;
+ /* IF (dst COND imm) JUMP off */
+ case BPF_JMP | BPF_JEQ | BPF_K:
+ case BPF_JMP | BPF_JGT | BPF_K:
+ case BPF_JMP | BPF_JGE | BPF_K:
+ case BPF_JMP | BPF_JNE | BPF_K:
+ case BPF_JMP | BPF_JSGT | BPF_K:
+ case BPF_JMP | BPF_JSGE | BPF_K:
+ if (is_simm13(imm)) {
+ emit_cmpi(dst, imm, ctx);
+ } else {
+ ctx->tmp_1_used = true;
+ emit_loadimm_sext(imm, bpf2sparc[TMP_REG_1], ctx);
+ emit_cmp(dst, bpf2sparc[TMP_REG_1], ctx);
+ }
+ goto emit_cond_jmp;
+ case BPF_JMP | BPF_JSET | BPF_K:
+ if (is_simm13(imm)) {
+ emit_btsti(dst, imm, ctx);
+ } else {
+ ctx->tmp_1_used = true;
+ emit_loadimm_sext(imm, bpf2sparc[TMP_REG_1], ctx);
+ emit_btst(dst, bpf2sparc[TMP_REG_1], ctx);
+ }
+ goto emit_cond_jmp;
+
+ /* function call */
+ case BPF_JMP | BPF_CALL:
+ {
+ u8 *func = ((u8 *)__bpf_call_base) + imm;
+
+ emit_call((u32 *)func, ctx);
+ emit_nop(ctx);
+
+ emit_reg_move(O0, bpf2sparc[BPF_REG_0], ctx);
+
+ if (bpf_helper_changes_pkt_data(func) && ctx->saw_ld_abs_ind)
+ load_skb_regs(ctx, bpf2sparc[BPF_REG_6]);
+ break;
+ }
+
+ /* function return */
+ case BPF_JMP | BPF_EXIT:
+ /* Optimization: when last instruction is EXIT,
+ simply fallthrough to epilogue. */
+ if (i == ctx->prog->len - 1)
+ break;
+ emit_branch(BA, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_nop(ctx);
+ break;
+
+ /* dst = imm64 */
+ case BPF_LD | BPF_IMM | BPF_DW:
+ {
+ const struct bpf_insn insn1 = insn[1];
+ u64 imm64;
+
+ imm64 = (u64)insn1.imm << 32 | (u32)imm;
+ emit_loadimm64(imm64, dst, ctx);
+
+ return 1;
+ }
+
+ /* LDX: dst = *(size *)(src + off) */
+ case BPF_LDX | BPF_MEM | BPF_W:
+ case BPF_LDX | BPF_MEM | BPF_H:
+ case BPF_LDX | BPF_MEM | BPF_B:
+ case BPF_LDX | BPF_MEM | BPF_DW: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ u32 opcode = 0, rs2;
+ s32 real_off = off;
+
+ ctx->tmp_1_used = true;
+ if (src == FP)
+ real_off += STACK_BIAS;
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ opcode = LD32;
+ break;
+ case BPF_H:
+ opcode = LD16;
+ break;
+ case BPF_B:
+ opcode = LD8;
+ break;
+ case BPF_DW:
+ opcode = LD64;
+ break;
+ }
+
+ if (is_simm13(real_off)) {
+ opcode |= IMMED;
+ rs2 = S13(real_off);
+ } else {
+ emit_loadimm(real_off, tmp, ctx);
+ rs2 = RS2(tmp);
+ }
+ emit(opcode | RS1(src) | rs2 | RD(dst), ctx);
+ break;
+ }
+ /* ST: *(size *)(dst + off) = imm */
+ case BPF_ST | BPF_MEM | BPF_W:
+ case BPF_ST | BPF_MEM | BPF_H:
+ case BPF_ST | BPF_MEM | BPF_B:
+ case BPF_ST | BPF_MEM | BPF_DW: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ const u8 tmp2 = bpf2sparc[TMP_REG_2];
+ u32 opcode = 0, rs2;
+ s32 real_off = off;
+
+ if (dst == FP)
+ real_off += STACK_BIAS;
+
+ ctx->tmp_2_used = true;
+ emit_loadimm(imm, tmp2, ctx);
+
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ opcode = ST32;
+ break;
+ case BPF_H:
+ opcode = ST16;
+ break;
+ case BPF_B:
+ opcode = ST8;
+ break;
+ case BPF_DW:
+ opcode = ST64;
+ break;
+ }
+
+ if (is_simm13(real_off)) {
+ opcode |= IMMED;
+ rs2 = S13(real_off);
+ } else {
+ ctx->tmp_1_used = true;
+ emit_loadimm(real_off, tmp, ctx);
+ rs2 = RS2(tmp);
+ }
+ emit(opcode | RS1(dst) | rs2 | RD(tmp2), ctx);
+ break;
+ }
+
+ /* STX: *(size *)(dst + off) = src */
+ case BPF_STX | BPF_MEM | BPF_W:
+ case BPF_STX | BPF_MEM | BPF_H:
+ case BPF_STX | BPF_MEM | BPF_B:
+ case BPF_STX | BPF_MEM | BPF_DW: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ u32 opcode = 0, rs2;
+ s32 real_off = off;
+
+ if (dst == FP)
+ real_off += STACK_BIAS;
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ opcode = ST32;
+ break;
+ case BPF_H:
+ opcode = ST16;
+ break;
+ case BPF_B:
+ opcode = ST8;
+ break;
+ case BPF_DW:
+ opcode = ST64;
+ break;
+ }
+ if (is_simm13(real_off)) {
+ opcode |= IMMED;
+ rs2 = S13(real_off);
+ } else {
+ ctx->tmp_1_used = true;
+ emit_loadimm(real_off, tmp, ctx);
+ rs2 = RS2(tmp);
+ }
+ emit(opcode | RS1(dst) | rs2 | RD(src), ctx);
+ break;
+ }
+
+ /* STX XADD: lock *(u32 *)(dst + off) += src */
+ case BPF_STX | BPF_XADD | BPF_W: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ const u8 tmp2 = bpf2sparc[TMP_REG_2];
+ const u8 tmp3 = bpf2sparc[TMP_REG_3];
+ s32 real_off = off;
+
+ ctx->tmp_1_used = true;
+ ctx->tmp_2_used = true;
+ ctx->tmp_3_used = true;
+ if (dst == FP)
+ real_off += STACK_BIAS;
+ emit_loadimm(real_off, tmp, ctx);
+ emit_alu3(ADD, dst, tmp, tmp, ctx);
+
+ emit(LD32 | RS1(tmp) | RS2(G0) | RD(tmp2), ctx);
+ emit_alu3(ADD, tmp2, src, tmp3, ctx);
+ emit(CAS | ASI(ASI_P) | RS1(tmp) | RS2(tmp2) | RD(tmp3), ctx);
+ emit_cmp(tmp2, tmp3, ctx);
+ emit_branch(BNE, 4, 0, ctx);
+ emit_nop(ctx);
+ break;
+ }
+ /* STX XADD: lock *(u64 *)(dst + off) += src */
+ case BPF_STX | BPF_XADD | BPF_DW: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ const u8 tmp2 = bpf2sparc[TMP_REG_2];
+ const u8 tmp3 = bpf2sparc[TMP_REG_3];
+ s32 real_off = off;
+
+ ctx->tmp_1_used = true;
+ ctx->tmp_2_used = true;
+ ctx->tmp_3_used = true;
+ if (dst == FP)
+ real_off += STACK_BIAS;
+ emit_loadimm(real_off, tmp, ctx);
+ emit_alu3(ADD, dst, tmp, tmp, ctx);
+
+ emit(LD64 | RS1(tmp) | RS2(G0) | RD(tmp2), ctx);
+ emit_alu3(ADD, tmp2, src, tmp3, ctx);
+ emit(CASX | ASI(ASI_P) | RS1(tmp) | RS2(tmp2) | RD(tmp3), ctx);
+ emit_cmp(tmp2, tmp3, ctx);
+ emit_branch(BNE, 4, 0, ctx);
+ emit_nop(ctx);
+ break;
+ }
+#define CHOOSE_LOAD_FUNC(K, func) \
+ ((int)K < 0 ? ((int)K >= SKF_LL_OFF ? func##_negative_offset : func) : func##_positive_offset)
+
+ /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + imm)) */
+ case BPF_LD | BPF_ABS | BPF_W:
+ func = CHOOSE_LOAD_FUNC(imm, bpf_jit_load_word);
+ goto common_load;
+ case BPF_LD | BPF_ABS | BPF_H:
+ func = CHOOSE_LOAD_FUNC(imm, bpf_jit_load_half);
+ goto common_load;
+ case BPF_LD | BPF_ABS | BPF_B:
+ func = CHOOSE_LOAD_FUNC(imm, bpf_jit_load_byte);
+ goto common_load;
+ /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + src + imm)) */
+ case BPF_LD | BPF_IND | BPF_W:
+ func = bpf_jit_load_word;
+ goto common_load;
+ case BPF_LD | BPF_IND | BPF_H:
+ func = bpf_jit_load_half;
+ goto common_load;
+
+ case BPF_LD | BPF_IND | BPF_B:
+ func = bpf_jit_load_byte;
+ common_load:
+ ctx->saw_ld_abs_ind = true;
+
+ emit_reg_move(bpf2sparc[BPF_REG_6], O0, ctx);
+ emit_loadimm(imm, O1, ctx);
+
+ if (BPF_MODE(code) == BPF_IND)
+ emit_alu(ADD, src, O1, ctx);
+
+ emit_call(func, ctx);
+ emit_alu_K(SRA, O1, 0, ctx);
+
+ emit_reg_move(O0, bpf2sparc[BPF_REG_0], ctx);
+ break;
+
+ default:
+ pr_err_once("unknown opcode %02x\n", code);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int build_body(struct jit_ctx *ctx)
+{
+ const struct bpf_prog *prog = ctx->prog;
+ int i;
+
+ for (i = 0; i < prog->len; i++) {
+ const struct bpf_insn *insn = &prog->insnsi[i];
+ int ret;
+
+ ret = build_insn(insn, ctx);
+ ctx->offset[i] = ctx->idx;
+
+ if (ret > 0) {
+ i++;
+ continue;
+ }
+ if (ret)
+ return ret;
+ }
+ return 0;
+}
+
+static void jit_fill_hole(void *area, unsigned int size)
+{
+ u32 *ptr;
+ /* We are guaranteed to have aligned memory. */
+ for (ptr = area; size >= sizeof(u32); size -= sizeof(u32))
+ *ptr++ = 0x91d02005; /* ta 5 */
+}
+
+struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+{
+ struct bpf_prog *tmp, *orig_prog = prog;
+ struct bpf_binary_header *header;
+ bool tmp_blinded = false;
+ struct jit_ctx ctx;
+ u32 image_size;
+ u8 *image_ptr;
+ int pass;
+
+ if (!bpf_jit_enable)
+ return orig_prog;
+
+ if (!prog || !prog->len)
+ return orig_prog;
+
+ tmp = bpf_jit_blind_constants(prog);
+ /* If blinding was requested and we failed during blinding,
+ * we must fall back to the interpreter.
+ */
+ if (IS_ERR(tmp))
+ return orig_prog;
+ if (tmp != prog) {
+ tmp_blinded = true;
+ prog = tmp;
+ }
+
+ memset(&ctx, 0, sizeof(ctx));
+ ctx.prog = prog;
+
+ ctx.offset = kcalloc(prog->len, sizeof(unsigned int), GFP_KERNEL);
+ if (ctx.offset == NULL) {
+ prog = orig_prog;
+ goto out;
+ }
+
+ /* Fake pass to detect features used, and get an accurate assessment
+ * of what the final image size will be.
+ */
+ if (build_body(&ctx)) {
+ prog = orig_prog;
+ goto out_off;
+ }
+ build_prologue(&ctx);
+ build_epilogue(&ctx);
+
+ /* Now we know the actual image size. */
+ image_size = sizeof(u32) * ctx.idx;
+ header = bpf_jit_binary_alloc(image_size, &image_ptr,
+ sizeof(u32), jit_fill_hole);
+ if (header == NULL) {
+ prog = orig_prog;
+ goto out_off;
+ }
+
+ ctx.image = (u32 *)image_ptr;
+
+ for (pass = 1; pass < 3; pass++) {
+ ctx.idx = 0;
+
+ build_prologue(&ctx);
+
+ if (build_body(&ctx)) {
+ bpf_jit_binary_free(header);
+ prog = orig_prog;
+ goto out_off;
+ }
+
+ build_epilogue(&ctx);
+
+ if (bpf_jit_enable > 1)
+ pr_info("Pass %d: shrink = %d, seen = [%c%c%c%c%c]\n", pass,
+ image_size - (ctx.idx * 4),
+ ctx.tmp_1_used ? '1' : ' ',
+ ctx.tmp_2_used ? '2' : ' ',
+ ctx.tmp_3_used ? '3' : ' ',
+ ctx.saw_ld_abs_ind ? 'L' : ' ',
+ ctx.saw_frame_pointer ? 'F' : ' ');
+ }
+
+ if (bpf_jit_enable > 1)
+ bpf_jit_dump(prog->len, image_size, pass, ctx.image);
+ bpf_flush_icache(ctx.image, ctx.image + image_size);
+
+ bpf_jit_binary_lock_ro(header);
+
+ prog->bpf_func = (void *)ctx.image;
+ prog->jited = 1;
+
+out_off:
+ kfree(ctx.offset);
+out:
+ if (tmp_blinded)
+ bpf_jit_prog_release_other(prog, prog == orig_prog ?
+ tmp : orig_prog);
+ return prog;
+}
--
2.1.2.532.g19b5d50
^ permalink raw reply related
* Re: [PATCH v2] smsc95xx: Use skb_cow_head to deal with cloned skbs
From: Eric Dumazet @ 2017-04-18 18:58 UTC (permalink / raw)
To: James Hughes; +Cc: netdev, Steve Glendinning, Microchip Linux Driver Support
In-Reply-To: <20170418171706.21190-1-james.hughes@raspberrypi.org>
On Tue, 2017-04-18 at 18:17 +0100, James Hughes wrote:
> The driver was failing to check that the SKB wasn't cloned
> before adding checksum data.
> Replace existing handling to extend/copy the header buffer
> with skb_cow_head.
>
> Signed-off-by: James Hughes <james.hughes@raspberrypi.org>
> ---
> Changes in v2
> - Changed skb_cow to skb_cow_head as suggested by netdev list
>
>
> drivers/net/usb/smsc95xx.c | 10 +++-------
> 1 file changed, 3 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/net/usb/smsc95xx.c b/drivers/net/usb/smsc95xx.c
> index df60c98..094f0ee 100644
> --- a/drivers/net/usb/smsc95xx.c
> +++ b/drivers/net/usb/smsc95xx.c
> @@ -2067,13 +2067,9 @@ static struct sk_buff *smsc95xx_tx_fixup(struct usbnet *dev,
> /* We do not advertise SG, so skbs should be already linearized */
> BUG_ON(skb_shinfo(skb)->nr_frags);
>
> - if (skb_headroom(skb) < overhead) {
> - struct sk_buff *skb2 = skb_copy_expand(skb,
> - overhead, 0, flags);
> - dev_kfree_skb_any(skb);
> - skb = skb2;
> - if (!skb)
> - return NULL;
> + /* Make writable and expand header space by overhead if required */
> + if (skb_cow_head(skb, overhead)) {
I believe you still need to
dev_kfree_skb_any(skb);
> + return NULL;
> }
>
> if (csum) {
^ permalink raw reply
* Re: question about size of sk_buff and skb_shared_info
From: Eric Dumazet @ 2017-04-18 19:00 UTC (permalink / raw)
To: Code Soldier1; +Cc: netdev
In-Reply-To: <CABGNecyP8tpBtbOPL_4fyhqnkTYEa80sHCuDrUJdm9gxihFX6w@mail.gmail.com>
On Tue, 2017-04-18 at 10:34 -0700, Code Soldier1 wrote:
> Hi Folks,
>
> I am sure there is a reason for the current sizes of these structures,
> However the reason is not obvious to me. So please help me understand.
>
> Currently the size of sk_buff on an x86_64 system is 232 bytes -- Why
> is that. I expected it to be a multiple of 32/64 as they are the most
> common cache lines. Since the alignment calculation will align the
> structure with the hw cache line, it seems like we might be wasting
> space ?
>
> skb_shared_info on the other hand is perfectly aligned with a size of 320 bytes.
>
> Thanks,
>
The alignment is there.
Look at skb_init() code, using SLAB_HWCACHE_ALIGN
^ permalink raw reply
* Re: NFS over NAT causes e1000e transmit hangs
From: Eric Dumazet @ 2017-04-18 19:03 UTC (permalink / raw)
To: Florian Fainelli; +Cc: intel-wired-lan, jeffrey.t.kirsher, netdev
In-Reply-To: <42af0e78-3107-1605-f8e1-d73a8c441ff0@gmail.com>
On Tue, 2017-04-18 at 11:18 -0700, Florian Fainelli wrote:
> Hi,
>
> I am using NFS over a NAT with two e1000e adapters and with eth1 being
> the LAN interface and eth0 the WAN interface. The kernel is Ubuntu's
> 16.10 kernel: 4.8.0-46-generic. The device doing NAT over NFS is just
> mounting a remote folder and doing normal execution/file accesses. It's
> enough to untar a file from this device onto a NFS share to expose the
> problem.
>
> The transmit hangs look like the ones below, doing a rmmod/insmod does
> not help eliminated the problem, nor does a power cycle. Stopping the
> NFS over NAT definitively does let the adapter recover.
Is this NFS over TCP or UDP ?
Thanks !
^ permalink raw reply
* Re: NFS over NAT causes e1000e transmit hangs
From: Florian Fainelli @ 2017-04-18 19:05 UTC (permalink / raw)
To: Eric Dumazet; +Cc: intel-wired-lan, jeffrey.t.kirsher, netdev
In-Reply-To: <1492542204.10587.138.camel@edumazet-glaptop3.roam.corp.google.com>
On 04/18/2017 12:03 PM, Eric Dumazet wrote:
> On Tue, 2017-04-18 at 11:18 -0700, Florian Fainelli wrote:
>> Hi,
>>
>> I am using NFS over a NAT with two e1000e adapters and with eth1 being
>> the LAN interface and eth0 the WAN interface. The kernel is Ubuntu's
>> 16.10 kernel: 4.8.0-46-generic. The device doing NAT over NFS is just
>> mounting a remote folder and doing normal execution/file accesses. It's
>> enough to untar a file from this device onto a NFS share to expose the
>> problem.
>>
>> The transmit hangs look like the ones below, doing a rmmod/insmod does
>> not help eliminated the problem, nor does a power cycle. Stopping the
>> NFS over NAT definitively does let the adapter recover.
>
> Is this NFS over TCP or UDP ?
This is NFS over TCP mounted with the following:
type nfs
(rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=2049,timeo=70,retrans=3,sec=sys,local_lock=none,addr=X.X.X.X)
Thanks Eric!
--
Florian
^ permalink raw reply
* Re: [PATCH v4 net-next RFC] net: Generic XDP
From: Andy Gospodarek @ 2017-04-18 19:05 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: David Miller, michael.chan, netdev, xdp-newbies
In-Reply-To: <20170415005949.GB73685@ast-mbp.thefacebook.com>
On Fri, Apr 14, 2017 at 05:59:51PM -0700, Alexei Starovoitov wrote:
> On Thu, Apr 13, 2017 at 04:23:15PM -0400, David Miller wrote:
> > +
> > + switch (act) {
> > + case XDP_TX:
> > + __skb_push(skb, skb->mac_len);
>
> s/skb->mac_len/mac_len/
>
I was away from my keyboard for a few days, but was able to get some
time to test this today.
When using this change above suggested by Alexei, XDP_DROP and XDP_TX
actions appear to work well with xdp1 and xdp2.
I'm seeing some rather odd behavior with xdp_tx_tunnel so it might be
good to hold off on committing this just yet.
At first glance, it looks like there is enough headroom for the new
frame, but the resulting packet data do not look right and I'm actually
seeing some data that may be left on the stack from a previous caller.
> > + HARD_TX_UNLOCK(dev, txq);
> > + if (free_skb) {
> > + trace_xdp_exception(dev, xdp_prog, XDP_TX);
> > + kfree_skb(skb);
>
> nice that you didn't forget to add trace_xdp_exception in this path :)
>
> Overall looks good to me and other than the minor nit in tx, i think it
> should work for programs already used with in-driver xdp.
> I'll test it next week unless people beat me to it.
>
^ permalink raw reply
* Re: [PATCH v4 net-next RFC] net: Generic XDP
From: David Miller @ 2017-04-18 19:07 UTC (permalink / raw)
To: andy; +Cc: alexei.starovoitov, michael.chan, netdev, xdp-newbies
In-Reply-To: <20170418190535.GG4730@C02RW35GFVH8.dhcp.broadcom.net>
From: Andy Gospodarek <andy@greyhouse.net>
Date: Tue, 18 Apr 2017 15:05:35 -0400
> On Fri, Apr 14, 2017 at 05:59:51PM -0700, Alexei Starovoitov wrote:
>> On Thu, Apr 13, 2017 at 04:23:15PM -0400, David Miller wrote:
>> > +
>> > + switch (act) {
>> > + case XDP_TX:
>> > + __skb_push(skb, skb->mac_len);
>>
>> s/skb->mac_len/mac_len/
>>
>
> I was away from my keyboard for a few days, but was able to get some
> time to test this today.
>
> When using this change above suggested by Alexei, XDP_DROP and XDP_TX
> actions appear to work well with xdp1 and xdp2.
>
> I'm seeing some rather odd behavior with xdp_tx_tunnel so it might be
> good to hold off on committing this just yet.
>
> At first glance, it looks like there is enough headroom for the new
> frame, but the resulting packet data do not look right and I'm actually
> seeing some data that may be left on the stack from a previous caller.
Thanks for testing Andy, I'll take a look and see if I can figure it out.
^ permalink raw reply
* [PATCH] dp83640: don't recieve time stamps twice
From: Dan Carpenter @ 2017-04-18 19:14 UTC (permalink / raw)
To: Richard Cochran, Stefan Sørensen
Cc: Andrew Lunn, Florian Fainelli, netdev, kernel-janitors
This patch is prompted by a static checker warning about a potential
use after free. The concern is that netif_rx_ni() can free "skb" and we
call it twice.
When I look at the commit that added this, it looks like some stray
lines were added accidentally. It doesn't make sense to me that we
would recieve the same data two times. I asked the author but never
recieved a response.
I can't test this code, but I'm pretty sure my patch is correct.
Fixes: 4b063258ab93 ("dp83640: Delay scheduled work.")
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
diff --git a/drivers/net/phy/dp83640.c b/drivers/net/phy/dp83640.c
index e2460a57e4b1..ed0d10f54f26 100644
--- a/drivers/net/phy/dp83640.c
+++ b/drivers/net/phy/dp83640.c
@@ -1438,8 +1438,6 @@ static bool dp83640_rxtstamp(struct phy_device *phydev,
skb_info->tmo = jiffies + SKB_TIMESTAMP_TIMEOUT;
skb_queue_tail(&dp83640->rx_queue, skb);
schedule_delayed_work(&dp83640->ts_work, SKB_TIMESTAMP_TIMEOUT);
- } else {
- netif_rx_ni(skb);
}
return true;
^ permalink raw reply related
* Re: [PATCH v4 net-next RFC] net: Generic XDP
From: David Miller @ 2017-04-18 19:29 UTC (permalink / raw)
To: andy; +Cc: alexei.starovoitov, michael.chan, netdev, xdp-newbies
In-Reply-To: <20170418.150708.1605529107204449972.davem@davemloft.net>
From: David Miller <davem@davemloft.net>
Date: Tue, 18 Apr 2017 15:07:08 -0400 (EDT)
> From: Andy Gospodarek <andy@greyhouse.net>
> Date: Tue, 18 Apr 2017 15:05:35 -0400
>
>> On Fri, Apr 14, 2017 at 05:59:51PM -0700, Alexei Starovoitov wrote:
>>> On Thu, Apr 13, 2017 at 04:23:15PM -0400, David Miller wrote:
>>> > +
>>> > + switch (act) {
>>> > + case XDP_TX:
>>> > + __skb_push(skb, skb->mac_len);
>>>
>>> s/skb->mac_len/mac_len/
>>>
>>
>> I was away from my keyboard for a few days, but was able to get some
>> time to test this today.
>>
>> When using this change above suggested by Alexei, XDP_DROP and XDP_TX
>> actions appear to work well with xdp1 and xdp2.
>>
>> I'm seeing some rather odd behavior with xdp_tx_tunnel so it might be
>> good to hold off on committing this just yet.
>>
>> At first glance, it looks like there is enough headroom for the new
>> frame, but the resulting packet data do not look right and I'm actually
>> seeing some data that may be left on the stack from a previous caller.
>
> Thanks for testing Andy, I'll take a look and see if I can figure it out.
Andy, I think we might be getting burnt by signedness issues in the
offset handling when the XDP program adjusts the packet data pointer.
In netif_receive_generic_xdp(), try changing the offset handling code to
read something like:
off = xdp.data - orig_data;
if (off > 0)
__skb_pull(skb, off);
else if (off < 0)
__skb_push(skb, -off);
If that doesn't work try adding:
__skb_cow(skb, XDP_PACKET_HEADROOM, 0);
right after the skb_linearize() call in that same function.
^ permalink raw reply
* Re: [PATCH net-next 1/2] sparc: Split BPF JIT into 32-bit and 64-bit.
From: Sam Ravnborg @ 2017-04-18 19:31 UTC (permalink / raw)
To: David Miller; +Cc: sparclinux, netdev, ast, daniel
In-Reply-To: <20170418.145806.845080704857445577.davem@davemloft.net>
On Tue, Apr 18, 2017 at 02:58:06PM -0400, David Miller wrote:
>
> This is in preparation for adding the 64-bit eBPF JIT.
>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> ---
> arch/sparc/net/Makefile | 2 +-
> arch/sparc/net/bpf_jit.h | 68 ----
> arch/sparc/net/bpf_jit_32.h | 68 ++++
> arch/sparc/net/bpf_jit_asm.S | 208 ----------
> arch/sparc/net/bpf_jit_asm_32.S | 208 ++++++++++
> arch/sparc/net/bpf_jit_asm_64.S | 1 +
> arch/sparc/net/bpf_jit_comp.c | 815 ---------------------------------------
> arch/sparc/net/bpf_jit_comp_32.c | 815 +++++++++++++++++++++++++++++++++++++++
> arch/sparc/net/bpf_jit_comp_64.c | 1 +
> 9 files changed, 1094 insertions(+), 1092 deletions(-)
> delete mode 100644 arch/sparc/net/bpf_jit.h
> create mode 100644 arch/sparc/net/bpf_jit_32.h
> delete mode 100644 arch/sparc/net/bpf_jit_asm.S
> create mode 100644 arch/sparc/net/bpf_jit_asm_32.S
> create mode 100644 arch/sparc/net/bpf_jit_asm_64.S
> delete mode 100644 arch/sparc/net/bpf_jit_comp.c
> create mode 100644 arch/sparc/net/bpf_jit_comp_32.c
> create mode 100644 arch/sparc/net/bpf_jit_comp_64.c
Adding options "-M -C" to git format-patch would have helped readability
of this patch.
As it is now we cannot see what changes were made to the varoious files
when they were renamed.
(I trust you have made no irrelevant but wanted to mention it anyway)
Sam
^ permalink raw reply
* Re: [PATCH net-next 1/2] sparc: Split BPF JIT into 32-bit and 64-bit.
From: David Miller @ 2017-04-18 19:35 UTC (permalink / raw)
To: sam; +Cc: sparclinux, netdev, ast, daniel
In-Reply-To: <20170418193157.GA29862@ravnborg.org>
From: Sam Ravnborg <sam@ravnborg.org>
Date: Tue, 18 Apr 2017 21:31:57 +0200
> On Tue, Apr 18, 2017 at 02:58:06PM -0400, David Miller wrote:
>>
>> This is in preparation for adding the 64-bit eBPF JIT.
>>
>> Signed-off-by: David S. Miller <davem@davemloft.net>
>> ---
>> arch/sparc/net/Makefile | 2 +-
>> arch/sparc/net/bpf_jit.h | 68 ----
>> arch/sparc/net/bpf_jit_32.h | 68 ++++
>> arch/sparc/net/bpf_jit_asm.S | 208 ----------
>> arch/sparc/net/bpf_jit_asm_32.S | 208 ++++++++++
>> arch/sparc/net/bpf_jit_asm_64.S | 1 +
>> arch/sparc/net/bpf_jit_comp.c | 815 ---------------------------------------
>> arch/sparc/net/bpf_jit_comp_32.c | 815 +++++++++++++++++++++++++++++++++++++++
>> arch/sparc/net/bpf_jit_comp_64.c | 1 +
>> 9 files changed, 1094 insertions(+), 1092 deletions(-)
>> delete mode 100644 arch/sparc/net/bpf_jit.h
>> create mode 100644 arch/sparc/net/bpf_jit_32.h
>> delete mode 100644 arch/sparc/net/bpf_jit_asm.S
>> create mode 100644 arch/sparc/net/bpf_jit_asm_32.S
>> create mode 100644 arch/sparc/net/bpf_jit_asm_64.S
>> delete mode 100644 arch/sparc/net/bpf_jit_comp.c
>> create mode 100644 arch/sparc/net/bpf_jit_comp_32.c
>> create mode 100644 arch/sparc/net/bpf_jit_comp_64.c
>
> Adding options "-M -C" to git format-patch would have helped readability
> of this patch.
> As it is now we cannot see what changes were made to the varoious files
> when they were renamed.
>
> (I trust you have made no irrelevant but wanted to mention it anyway)
My bad, I'll remember to do this next time:
Since it's so small, here it is for reference:
====================
>From c651a5d37afed8461adc3f2824cae1d7facd2e7a Mon Sep 17 00:00:00 2001
From: "David S. Miller" <davem@davemloft.net>
Date: Mon, 17 Apr 2017 18:25:07 -0700
Subject: [PATCH 1/2] sparc: Split BPF JIT into 32-bit and 64-bit.
This is in preparation for adding the 64-bit eBPF JIT.
Signed-off-by: David S. Miller <davem@davemloft.net>
---
arch/sparc/net/Makefile | 2 +-
arch/sparc/net/{bpf_jit.h => bpf_jit_32.h} | 0
arch/sparc/net/{bpf_jit_asm.S => bpf_jit_asm_32.S} | 2 +-
arch/sparc/net/bpf_jit_asm_64.S | 1 +
arch/sparc/net/{bpf_jit_comp.c => bpf_jit_comp_32.c} | 2 +-
arch/sparc/net/bpf_jit_comp_64.c | 1 +
6 files changed, 5 insertions(+), 3 deletions(-)
rename arch/sparc/net/{bpf_jit.h => bpf_jit_32.h} (100%)
rename arch/sparc/net/{bpf_jit_asm.S => bpf_jit_asm_32.S} (99%)
create mode 100644 arch/sparc/net/bpf_jit_asm_64.S
rename arch/sparc/net/{bpf_jit_comp.c => bpf_jit_comp_32.c} (99%)
create mode 100644 arch/sparc/net/bpf_jit_comp_64.c
diff --git a/arch/sparc/net/Makefile b/arch/sparc/net/Makefile
index 1306a58..76fa8e9 100644
--- a/arch/sparc/net/Makefile
+++ b/arch/sparc/net/Makefile
@@ -1,4 +1,4 @@
#
# Arch-specific network modules
#
-obj-$(CONFIG_BPF_JIT) += bpf_jit_asm.o bpf_jit_comp.o
+obj-$(CONFIG_BPF_JIT) += bpf_jit_asm_$(BITS).o bpf_jit_comp_$(BITS).o
diff --git a/arch/sparc/net/bpf_jit.h b/arch/sparc/net/bpf_jit_32.h
similarity index 100%
rename from arch/sparc/net/bpf_jit.h
rename to arch/sparc/net/bpf_jit_32.h
diff --git a/arch/sparc/net/bpf_jit_asm.S b/arch/sparc/net/bpf_jit_asm_32.S
similarity index 99%
rename from arch/sparc/net/bpf_jit_asm.S
rename to arch/sparc/net/bpf_jit_asm_32.S
index 8c83f4b..5632cdc 100644
--- a/arch/sparc/net/bpf_jit_asm.S
+++ b/arch/sparc/net/bpf_jit_asm_32.S
@@ -1,6 +1,6 @@
#include <asm/ptrace.h>
-#include "bpf_jit.h"
+#include "bpf_jit_32.h"
#ifdef CONFIG_SPARC64
#define SAVE_SZ 176
diff --git a/arch/sparc/net/bpf_jit_asm_64.S b/arch/sparc/net/bpf_jit_asm_64.S
new file mode 100644
index 0000000..6fb023f
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_asm_64.S
@@ -0,0 +1 @@
+#include "bpf_jit_asm_32.S"
diff --git a/arch/sparc/net/bpf_jit_comp.c b/arch/sparc/net/bpf_jit_comp_32.c
similarity index 99%
rename from arch/sparc/net/bpf_jit_comp.c
rename to arch/sparc/net/bpf_jit_comp_32.c
index a6d9204..83fc41d 100644
--- a/arch/sparc/net/bpf_jit_comp.c
+++ b/arch/sparc/net/bpf_jit_comp_32.c
@@ -8,7 +8,7 @@
#include <asm/cacheflush.h>
#include <asm/ptrace.h>
-#include "bpf_jit.h"
+#include "bpf_jit_32.h"
int bpf_jit_enable __read_mostly;
diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c
new file mode 100644
index 0000000..49b5f65
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_comp_64.c
@@ -0,0 +1 @@
+#include "bpf_jit_comp_32.c"
--
2.1.2.532.g19b5d50
^ permalink raw reply related
* Re: [PATCH v4 net-next RFC] net: Generic XDP
From: Andy Gospodarek @ 2017-04-18 19:37 UTC (permalink / raw)
To: David Miller; +Cc: alexei.starovoitov, michael.chan, netdev, xdp-newbies
In-Reply-To: <20170418.152916.1361453741909754079.davem@davemloft.net>
On Tue, Apr 18, 2017 at 03:29:16PM -0400, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Tue, 18 Apr 2017 15:07:08 -0400 (EDT)
>
> > From: Andy Gospodarek <andy@greyhouse.net>
> > Date: Tue, 18 Apr 2017 15:05:35 -0400
> >
> >> On Fri, Apr 14, 2017 at 05:59:51PM -0700, Alexei Starovoitov wrote:
> >>> On Thu, Apr 13, 2017 at 04:23:15PM -0400, David Miller wrote:
> >>> > +
> >>> > + switch (act) {
> >>> > + case XDP_TX:
> >>> > + __skb_push(skb, skb->mac_len);
> >>>
> >>> s/skb->mac_len/mac_len/
> >>>
> >>
> >> I was away from my keyboard for a few days, but was able to get some
> >> time to test this today.
> >>
> >> When using this change above suggested by Alexei, XDP_DROP and XDP_TX
> >> actions appear to work well with xdp1 and xdp2.
> >>
> >> I'm seeing some rather odd behavior with xdp_tx_tunnel so it might be
> >> good to hold off on committing this just yet.
> >>
> >> At first glance, it looks like there is enough headroom for the new
> >> frame, but the resulting packet data do not look right and I'm actually
> >> seeing some data that may be left on the stack from a previous caller.
> >
> > Thanks for testing Andy, I'll take a look and see if I can figure it out.
>
> Andy, I think we might be getting burnt by signedness issues in the
> offset handling when the XDP program adjusts the packet data pointer.
I completely agree -- I just noted that the offset would be -20 in the
tx_tunnel case and it's easy to get confused since a positive int for
the second arg in skb_pull() does go 'back' with a positive value and
was just rebuilding and testing just this case with:
- off = xdp.data - orig_data;
+ /* note that offset is negative */
+ off = orig_data - xdp.data;
> In netif_receive_generic_xdp(), try changing the offset handling code to
> read something like:
>
> off = xdp.data - orig_data;
> if (off > 0)
> __skb_pull(skb, off);
> else if (off < 0)
> __skb_push(skb, -off);
This should do it. I'll give that run, too.
> If that doesn't work try adding:
>
> __skb_cow(skb, XDP_PACKET_HEADROOM, 0);
>
> right after the skb_linearize() call in that same function.
^ permalink raw reply
* FEC on i.MX 7 transmit queue timeout
From: Stefan Agner @ 2017-04-18 19:46 UTC (permalink / raw)
To: fugang.duan, festevam; +Cc: netdev
Hi,
I noticed last week on upstream (v4.11-rc6) on a Colibri iMX7 board that
after a while (~10 minutes) the detdev wachdog prints a stacktrace and
the driver then continuously dumps the TX ring. I then did a quick test
with 4.10, and realized it actually suffers the same issue, so it seems
not to be a regression. I use a rootfs mounted over NFS...
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
dev_watchdog+0x240/0x244
NETDEV WATCHDOG: eth0 (fec): transmit queue 2 timed out
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted
4.11.0-rc7-00030-g2c4e6bd0c4f0-dirty #330
Hardware name: Freescale i.MX7 Dual (Device Tree)
[<c02293f0>] (unwind_backtrace) from [<c0225820>] (show_stack+0x10/0x14)
[<c0225820>] (show_stack) from [<c050db6c>] (dump_stack+0x90/0xa0)
[<c050db6c>] (dump_stack) from [<c023ae68>] (__warn+0xac/0x11c)
[<c023ae68>] (__warn) from [<c023af10>] (warn_slowpath_fmt+0x38/0x48)
[<c023af10>] (warn_slowpath_fmt) from [<c088bb8c>]
(dev_watchdog+0x240/0x244)
[<c088bb8c>] (dev_watchdog) from [<c0294798>]
(run_timer_softirq+0x24c/0x708)
[<c0294798>] (run_timer_softirq) from [<c023f584>]
(__do_softirq+0x12c/0x2a8)
[<c023f584>] (__do_softirq) from [<c023f8c4>] (irq_exit+0xdc/0x13c)
[<c023f8c4>] (irq_exit) from [<c02818ac>]
(__handle_domain_irq+0xa4/0xf8)
[<c02818ac>] (__handle_domain_irq) from [<c0201624>]
(gic_handle_irq+0x34/0xa4)
[<c0201624>] (gic_handle_irq) from [<c0226338>] (__irq_svc+0x58/0x8c)
Exception stack(0xc1201f30 to 0xc1201f78)
1f20: c0233320 00000000 00000000
01400000
1f40: c1203d80 ffffe000 00000000 00000000 c107bf10 c0e055b5 c1203d34
00000001
1f60: c07d2324 c1201f80 c0222ac8 c0222acc 60000013 ffffffff
[<c0226338>] (__irq_svc) from [<c0222acc>] (arch_cpu_idle+0x38/0x3c)
[<c0222acc>] (arch_cpu_idle) from [<c0275f24>] (do_idle+0xa8/0x250)
[<c0275f24>] (do_idle) from [<c02760e4>] (cpu_startup_entry+0x18/0x1c)
[<c02760e4>] (cpu_startup_entry) from [<c1000aa0>]
(start_kernel+0x3fc/0x45c)
---[ end trace 5b0c6dc3466a7918 ]---
fec 30be0000.ethernet eth0: TX ring dump
Nr SC addr len SKB
0 0x1c00 0x00000000 590 (null)
1 0x1c00 0x00000000 590 (null)
2 0x1c00 0x00000000 42 (null)
3 H 0x1c00 0x00000000 42 (null)
4 S 0x0000 0x00000000 0 (null)
5 0x0000 0x00000000 0 (null)
6 0x0000 0x00000000 0 (null)
7 0x0000 0x00000000 0 (null)
8 0x0000 0x00000000 0 (null)
9 0x0000 0x00000000 0 (null)
10 0x0000 0x00000000 0 (null)
11 0x0000 0x00000000 0 (null)
12 0x0000 0x00000000 0 (null)
13 0x0000 0x00000000 0 (null)
14 0x0000 0x00000000 0 (null)
15 0x0000 0x00000000 0 (null)
16 0x0000 0x00000000 0 (null)
17 0x0000 0x00000000 0 (null)
18 0x0000 0x00000000 0 (null)
...
A second TX ring dump from 4.10:
fec 30be0000.ethernet eth0: TX ring dump
Nr SC addr len SKB
0 0x1c00 0x00000000 42 (null)
1 0x1c00 0x00000000 42 (null)
2 0x1c00 0x00000000 90 (null)
3 0x1c00 0x00000000 90 (null)
4 0x1c00 0x00000000 90 (null)
5 0x1c00 0x00000000 218 (null)
6 0x1c00 0x00000000 218 (null)
7 0x1c00 0x00000000 218 (null)
8 0x1c00 0x00000000 90 (null)
9 0x1c00 0x00000000 206 (null)
10 0x1c00 0x00000000 216 (null)
11 0x1c00 0x00000000 216 (null)
12 0x1c00 0x00000000 216 (null)
13 0x1c00 0x00000000 311 (null)
14 0x1c00 0x00000000 178 (null)
15 0x1c00 0x00000000 311 (null)
16 0x1c00 0x00000000 206 (null)
17 H 0x1c00 0x00000000 311 (null)
18 S 0x0000 0x00000000 0 (null)
19 0x0000 0x00000000 0 (null)
The ring dump prints continously, but I can access console every now and
then. I noticed that the second interrupt seems static (66441, TX
interrupt?):
58: 18 GIC-0 150 Level 30be0000.ethernet
59: 66441 GIC-0 151 Level 30be0000.ethernet
60: 70477 GIC-0 152 Level 30be0000.ethernet
Anybody else seen this? Any idea?
In 4.10 as well as 4.11-rc6 the interrupt counts were just over 65536...
pure chance?
^ permalink raw reply
* Re: [PATCH v2 net-next] drivers: net: xgene-v2: Extend ethtool statistics
From: David Miller @ 2017-04-18 19:55 UTC (permalink / raw)
To: isubramanian; +Cc: netdev, linux-arm-kernel, patches, kchudgar
In-Reply-To: <1492472875-32385-1-git-send-email-isubramanian@apm.com>
From: Iyappan Subramanian <isubramanian@apm.com>
Date: Mon, 17 Apr 2017 16:47:55 -0700
> This patch adds extended statistics reporting to ethtool.
>
> In summary, this patch,
>
> - adds ethtool.h with the statistics register definitions
> - adds 'struct xge_gstrings_extd_stats' to gather extended stats
> - modifies xge_get_strings(), get_sset_count() and
> get_ethtool_stats() accordingly
> - moves 'struct xge_gstrings_stats' to ethtool.h
>
> Signed-off-by: Iyappan Subramanian <isubramanian@apm.com>
Applied, thanks.
^ permalink raw reply
* Re: [PATCH net] be2net: VxLAN offload should be re-enabled when only 1 UDP port is left
From: David Miller @ 2017-04-18 19:56 UTC (permalink / raw)
To: sriharsha.basavapatna; +Cc: netdev
In-Reply-To: <20170417160313.1548-1-sriharsha.basavapatna@broadcom.com>
From: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
Date: Mon, 17 Apr 2017 21:33:13 +0530
> We disable VxLAN offload when more than 1 UDP port is added to the driver,
> since Skyhawk doesn't support offload with multiple ports. The existing
> driver design expects the user to delete all port configurations and create
> a configuration with a single UDP port for VxLAN offload to be re-enabled.
> Remove this restriction by tracking the ports added and re-enabling offload
> when ports get deleted and only 1 port is left.
>
> Signed-off-by: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
> Reviewed-by: Sathya Perla <sathya.perla@broadcom.com>
Applied to net-next, thanks.
^ permalink raw reply
* XDP question: best API for returning/setting egress port?
From: Jesper Dangaard Brouer @ 2017-04-18 19:58 UTC (permalink / raw)
To: Daniel Borkmann, Alexei Starovoitov, Alexei Starovoitov
Cc: brouer, netdev@vger.kernel.org, xdp-newbies@vger.kernel.org,
John Fastabend
As I argued in NetConf presentation[1] (from slide #9) we need a port
mapping table (instead of using ifindex'es). Both for supporting
other "port" types than net_devices (think sockets), and for
sandboxing what XDP can bypass.
I want to create a new XDP action called XDP_REDIRECT, that instruct
XDP to send the xdp_buff to another "port" (get translated into a
net_device, or something else depending on internal port type).
Looking at the userspace/eBPF interface, I'm wondering what is the
best API for "returning" this port number from eBPF?
The options I see is:
1) Split-up the u32 action code, and e.g let the high-16-bit be the
port number and lower-16bit the (existing) action verdict.
Pros: Simple API
Cons: Number of ports limited to 64K
2) Extend both xdp_buff + xdp_md to contain a (u32) port number, allow
eBPF to update xdp_md->port.
Pros: Larger number of ports.
Cons: This require some ebpf translation steps between xdp_buff <-> xdp_md.
(see xdp_convert_ctx_access)
3) Extend only xdp_buff and create bpf_helper that set port in xdp_buff.
Pros: Hides impl details, and allows helper to give eBPF code feedback
(on e.g. if port doesn't exist any longer)
Cons: Helper function call likely slower?
(Cc'ed xdp-newbies as end-users might have an opinion on UAPI?)
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
[1] http://people.netfilter.org/hawk/presentations/NetConf2017/xdp_work_ahead_NetConf_April_2017.pdf
^ permalink raw reply
* Re: [PATCH v2] net: cx89x0: move attribute declaration before struct keyword
From: David Miller @ 2017-04-18 20:00 UTC (permalink / raw)
To: stefan; +Cc: shc_work, jarod, netdev, linux-kernel
In-Reply-To: <20170417205434.12663-1-stefan@agner.ch>
From: Stefan Agner <stefan@agner.ch>
Date: Mon, 17 Apr 2017 13:54:34 -0700
> The attribute declaration is typically before the definition. Move
> the __maybe_unused attribute declaration before the struct keyword.
>
> Signed-off-by: Stefan Agner <stefan@agner.ch>
> ---
> Changes in v2:
> - Move __maybe_unused after the complete type
Applied to net-next, thank you.
^ permalink raw reply
* Re: [PATCH][net-next] esp6: fix incorrect null pointer check on xo
From: Dan Carpenter @ 2017-04-18 20:11 UTC (permalink / raw)
To: Colin King
Cc: David S . Miller, Alexey Kuznetsov, James Morris,
Hideaki YOSHIFUJI, Patrick McHardy, netdev, kernel-janitors,
linux-kernel
In-Reply-To: <20170418140653.8839-1-colin.king@canonical.com>
It's in Steffen's tree, not the networking tree and he's not CC'd. You
should really be adding Fixes tags because it helps to ensure you CC the
guilty parties.
regards,
dan carpenter
^ permalink raw reply
* Re: [PATCH] dp83640: don't recieve time stamps twice
From: Richard Cochran @ 2017-04-18 20:19 UTC (permalink / raw)
To: Dan Carpenter
Cc: Stefan Sørensen, Andrew Lunn, Florian Fainelli, netdev,
kernel-janitors
In-Reply-To: <20170418191426.GA17838@mwanda>
On Tue, Apr 18, 2017 at 10:14:26PM +0300, Dan Carpenter wrote:
> This patch is prompted by a static checker warning about a potential
> use after free. The concern is that netif_rx_ni() can free "skb" and we
> call it twice.
Right, the code already calls netif_rx_ni() in the list_for_each_safe()
loop just above, in the case that the shhwtstamps pointer has been set.
> When I look at the commit that added this, it looks like some stray
> lines were added accidentally. It doesn't make sense to me that we
> would recieve the same data two times. I asked the author but never
> recieved a response.
Hm, maybe the intent was to move the call to netif_rx_ni() outside of
the spin_lock_irqsave() region (which how I had it before Stefan's
changes).
But calling netif_rx_ni() twice is clearly wrong.
Thanks,
Richard
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox