* Re: [PATCH] net: stmmac: add dcrs parameter
From: Chen-Yu Tsai @ 2014-08-27 2:36 UTC (permalink / raw)
To: Giuseppe CAVALLARO
Cc: Vince Bridgers, Ley Foon Tan, LeyFoon Tan, netdev, linux-kernel,
David S. Miller, Vince Bridgers
In-Reply-To: <53FC89A0.8040905@st.com>
On Tue, Aug 26, 2014 at 9:20 PM, Giuseppe CAVALLARO
<peppe.cavallaro@st.com> wrote:
> On 8/26/2014 2:35 PM, Vince Bridgers wrote:
>>
>> Hi Peppe,
>>
>>>>
>>>> In the Synopsys EMAC case, carrier sense is used to stop transmitting
>>>> if no carrier is sensed during a transmission. This is only useful if
>>>> the media in use is true half duplex media (like obsolete 10Base2 or
>>>> 10Base5). If no one in using true half duplex media, then is it
>>>> possible to set this disable by default? If we're not sure, then
>>>> having an option feels like the right thing to do.
>>>
>>>
>>>
>>> Indeed this is what I had done in the patch.
>>>
>>>
>>> http://git.stlinux.com/?p=stm/linux-sh4-2.6.32.y.git;a=commit;h=b0b863bf65c36dc593f6b7b4b418394fd880dae2
>>>
>>> Also in case of carrier sense the frame will be dropped in any case
>>> later.
>>>
>>> Let me know if you Acked this patch so I will rebase it on
>>> net.git and I send it soon
>>>
>>> peppe
>>>
>>
>> Yes, this looks good to me. I don't expect anyone is using 10Base2 or
>> 10Base5 anymore, so it's ok to disable DCRS by default.
>>
>> ack
>>
>> All the best,
>
>
> thx so much, I will send this patch (with your Acked-by) and ported on
> net.git soon.
>
> Chen-Yu, Ley Foon, pls let me know if it is ok for you as well
Looks good. Thanks!
Cheers
ChenYu
^ permalink raw reply
* Re: [patch net-next RFC 03/12] net: introduce generic switch devices support
From: Tom Herbert @ 2014-08-27 2:45 UTC (permalink / raw)
To: Ben Hutchings
Cc: ryazanov.s.a-Re5JQEeQqe8AvxtiuMwx3w, Jason Wang, John Fastabend,
Neil.Jerram-QnUH15yq9NYqDJ6do+/SaQ, Eric Dumazet,
andy-QlMahl40kYEqcZcGjlUOXw, dev-yBygre7rU0TnMu66kgdUjQ,
Felix Fietkau, Florian Fainelli, ronye-VPRAkNaXOzVWk0Htik3J/w,
Jeff Kirsher, Or Gerlitz, Lennert Buytenhek, Jiri Pirko,
roopa-qUQiAmfTcIp+XZJcv9eMoEEOCMrvLtNR, Jamal Hadi Salim,
aviadr-VPRAkNaXOzVWk0Htik3J/w, Nicolas Dichtel,
vyasevic-H+wXaHxf7aLQT0dZR+AlfA, Neil Horman, Linux Netdev List,
Stephen Hemminger, Daniel Borkmann, Eric W. Biederman,
David Miller
In-Reply-To: <1408639283.13073.3.camel-nDn/Rdv9kqW9Jme8/bJn5UCKIB8iOfG2tUK59QYPAWc@public.gmane.org>
On Thu, Aug 21, 2014 at 9:41 AM, Ben Hutchings <ben-/+tVBieCtBitmTQ+vhA3Yw@public.gmane.org> wrote:
> On Thu, 2014-08-21 at 18:18 +0200, Jiri Pirko wrote:
>> The goal of this is to provide a possibility to suport various switch
>> chips. Drivers should implement relevant ndos to do so. Now there is a
>> couple of ndos defines:
>> - for getting physical switch id is in place.
>> - for work with flows.
>>
>> Note that user can use random port netdevice to access the switch.
> [...]
>
> Why isn't the switch treated as a real device (not necessarily a net
> device) that's included in the device model and that the port devices
> refer to?
>
+1. In the same way that software switch (e.g. OVS) is mostly
implemented out of the core stack, I think hardware switch should have
same model. Besides, if you define this as part of a network interface
then you'll need to resolve unpleasant issues like accounting for
bytes and packets switched (e.g. are these accounted in interface
stats?)), what happens if netdevice is put in promiscuous mode (e.g.
do we need a tap for all switched packets?), etc. Another thing to
consider is that some vendors will undoubtably implement an open flow
agent in the NIC to manage the switch, possibly even autonomously from
the host-- a host might see the device but not be able to manage it
other than adding interfaces as local ports.
> Ben.
>
> --
> Ben Hutchings
> If at first you don't succeed, you're doing about average.
^ permalink raw reply
* Re: [PATCH net v3 4/4] tg3: Fix tx_pending checks for tg3_tso_bug
From: Prashant Sreedharan @ 2014-08-27 3:40 UTC (permalink / raw)
To: Benjamin Poirier; +Cc: Michael Chan, netdev, linux-kernel
In-Reply-To: <1409081178-4877-4-git-send-email-bpoirier@suse.de>
> static inline bool tg3_maybe_stop_txq(struct tg3_napi *tnapi,
> struct netdev_queue *txq,
> @@ -7841,14 +7847,16 @@ static inline bool tg3_maybe_stop_txq(struct tg3_napi *tnapi,
> if (!netif_tx_queue_stopped(txq)) {
> stopped = true;
> netif_tx_stop_queue(txq);
> - BUG_ON(wakeup_thresh >= tnapi->tx_pending);
> + tnapi->wakeup_thresh = wakeup_thresh;
> + BUG_ON(tnapi->wakeup_thresh >= tnapi->tx_pending);
> }
> /* netif_tx_stop_queue() must be done before checking tx index
> * in tg3_tx_avail(), because in tg3_tx(), we update tx index
> - * before checking for netif_tx_queue_stopped().
> + * before checking for netif_tx_queue_stopped(). The memory
> + * barrier also synchronizes wakeup_thresh changes.
> */
> smp_mb();
> - if (tg3_tx_avail(tnapi) > wakeup_thresh)
> + if (tg3_tx_avail(tnapi) > tnapi->wakeup_thresh)
> netif_tx_wake_queue(txq);
you can add a comment here... stopped is not set to false even if queue
wakes up, to log the netdev_err "BUG! TX Ring.." message.
> }
> return stopped;
> @@ -7861,10 +7869,10 @@ static int tg3_tso_bug(struct tg3 *tp, struct tg3_napi *tnapi,
> struct netdev_queue *txq, struct sk_buff *skb)
> @@ -12318,9 +12354,7 @@ static int tg3_set_ringparam(struct net_device *dev, struct ethtool_ringparam *e
> if ((ering->rx_pending > tp->rx_std_ring_mask) ||
> (ering->rx_jumbo_pending > tp->rx_jmb_ring_mask) ||
> (ering->tx_pending > TG3_TX_RING_SIZE - 1) ||
> - (ering->tx_pending <= MAX_SKB_FRAGS + 1) ||
> - (tg3_flag(tp, TSO_BUG) &&
> - (ering->tx_pending <= (MAX_SKB_FRAGS * 3))))
> + (ering->tx_pending <= MAX_SKB_FRAGS + 1))
> return -EINVAL;
>
> if (netif_running(dev)) {
> @@ -12340,6 +12374,7 @@ static int tg3_set_ringparam(struct net_device *dev, struct ethtool_ringparam *e
> if (tg3_flag(tp, JUMBO_RING_ENABLE))
> tp->rx_jumbo_pending = ering->rx_jumbo_pending;
>
> + dev->gso_max_segs = TG3_TX_SEG_PER_DESC(ering->tx_pending - 1);
Assuming a LSO skb of 64k size takes the tg3_tso_bug() code path, if the
available TX descriptors is <= 135 assuming gso_segs is 45 for this skb
based on the estimate 45 * 3 driver would stop this TX queue and set the
tnapi->wakeup_thresh to 135 and return NETDEV_TX_BUSY. This skb will be
queued to be resent when the queue wakes up.
Meanwhile if the user changes the TX ring size tx_pending=135,
dev->gso_max_segs is modified accordingly to 44, the LSO skb which was
queued will now be GSO'ed (in net/dev.c) before calling
tg3_start_xmit(). To note tg3_tx() cannot wake the queue as it is
expecting to be woken up when available free TX descriptors is 136. So
we end up with HW TX ring empty and not able to send any pkts.
> for (i = 0; i < tp->irq_max; i++)
> tp->napi[i].tx_pending = ering->tx_pending;
>
> @@ -17816,6 +17851,7 @@ static int tg3_init_one(struct pci_dev *pdev,
> else
^ permalink raw reply
* [net-next v3 1/3] udp: Expand UDP tunnel common APIs
From: Andy Zhou @ 2014-08-27 3:35 UTC (permalink / raw)
To: davem; +Cc: netdev, Andy Zhou
Added create_udp_tunnel_socket(), packet receive and transmit, and
other related common functions for UDP tunnels.
Per net open UDP tunnel ports are tracked in this common layer to
prevent sharing of a single port with more than one UDP tunnel.
Signed-off-by: Andy Zhou <azhou@nicira.com>
---
include/net/udp_tunnel.h | 50 +++++++++++
net/ipv4/udp_tunnel.c | 216 +++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 265 insertions(+), 1 deletion(-)
diff --git a/include/net/udp_tunnel.h b/include/net/udp_tunnel.h
index ffd69cb..5ff0746 100644
--- a/include/net/udp_tunnel.h
+++ b/include/net/udp_tunnel.h
@@ -1,6 +1,8 @@
#ifndef __NET_UDP_TUNNEL_H
#define __NET_UDP_TUNNEL_H
+#include <net/ip_tunnels.h>
+
struct udp_port_cfg {
u8 family;
@@ -26,7 +28,55 @@ struct udp_port_cfg {
use_udp6_rx_checksums:1;
};
+struct udp_tunnel_sock;
+
+typedef int (*udp_tunnel_encap_rcv_t)(struct sock *sk, struct sk_buff *skb);
+typedef void (*udp_tunnel_encap_destroy_t)(struct sock *sk);
+
+struct udp_tunnel_socket_cfg {
+ struct udp_port_cfg port;
+
+ /* Used for setting up udp_sock fields, see udp.h for details */
+ __u8 encap_type;
+ udp_tunnel_encap_rcv_t encap_rcv;
+ udp_tunnel_encap_destroy_t encap_destroy;
+};
+
+struct udp_tunnel_sock {
+ struct hlist_node hlist;
+ struct socket *sock;
+};
+
int udp_sock_create(struct net *net, struct udp_port_cfg *cfg,
struct socket **sockp);
+struct udp_tunnel_sock *create_udp_tunnel_sock(struct net *net, size_t size,
+ struct socket *sock,
+ struct udp_tunnel_socket_cfg
+ *socket_cfg);
+
+struct udp_tunnel_sock *udp_tunnel_find_sock(struct net *net, __be16 port);
+
+int udp_tunnel_xmit_skb(struct socket *sock, struct rtable *rt,
+ struct sk_buff *skb, __be32 src, __be32 dst,
+ __u8 tos, __u8 ttl, __be16 df, __be16 src_port,
+ __be16 dst_port, bool xnet);
+
+#if IS_ENABLED(CONFIG_IPV6)
+int udp_tunnel6_xmit_skb(struct socket *sock, struct dst_entry *dst,
+ struct sk_buff *skb, struct net_device *dev,
+ struct in6_addr *saddr, struct in6_addr *daddr,
+ __u8 prio, __u8 ttl, __be16 src_port, __be16 dst_port);
+
+#endif
+
+void udp_tunnel_sock_release(struct udp_tunnel_sock *uts);
+
+static inline struct sk_buff *udp_tunnel_handle_offloads(struct sk_buff *skb,
+ bool udp_csum)
+{
+ int type = udp_csum ? SKB_GSO_UDP_TUNNEL_CSUM : SKB_GSO_UDP_TUNNEL;
+
+ return iptunnel_handle_offloads(skb, udp_csum, type);
+}
#endif
diff --git a/net/ipv4/udp_tunnel.c b/net/ipv4/udp_tunnel.c
index 61ec1a6..3ed66bc 100644
--- a/net/ipv4/udp_tunnel.c
+++ b/net/ipv4/udp_tunnel.c
@@ -7,6 +7,23 @@
#include <net/udp.h>
#include <net/udp_tunnel.h>
#include <net/net_namespace.h>
+#include <net/netns/generic.h>
+#if IS_ENABLED(CONFIG_IPV6)
+#include <net/ipv6.h>
+#include <net/addrconf.h>
+#include <net/ip6_tunnel.h>
+#include <net/ip6_checksum.h>
+#endif
+
+#define PORT_HASH_BITS 8
+#define PORT_HASH_SIZE (1 << PORT_HASH_BITS)
+
+static int udp_tunnel_net_id;
+
+struct udp_tunnel_net {
+ struct hlist_head sock_list[PORT_HASH_SIZE];
+ spinlock_t sock_lock; /* Protecting the sock_list */
+};
int udp_sock_create(struct net *net, struct udp_port_cfg *cfg,
struct socket **sockp)
@@ -82,7 +99,6 @@ int udp_sock_create(struct net *net, struct udp_port_cfg *cfg,
return -EPFNOSUPPORT;
}
-
*sockp = sock;
return 0;
@@ -97,4 +113,202 @@ error:
}
EXPORT_SYMBOL(udp_sock_create);
+
+/* Socket hash table head */
+static inline struct hlist_head *uts_head(struct net *net, const __be16 port)
+{
+ struct udp_tunnel_net *utn = net_generic(net, udp_tunnel_net_id);
+
+ return &utn->sock_list[hash_32(ntohs(port), PORT_HASH_BITS)];
+}
+
+struct udp_tunnel_sock *create_udp_tunnel_sock(struct net *net, size_t size,
+ struct socket *sock,
+ struct udp_tunnel_socket_cfg
+ *cfg)
+{
+ struct udp_tunnel_net *utn = net_generic(net, udp_tunnel_net_id);
+ struct udp_tunnel_sock *uts;
+ struct sock *sk;
+ const __be16 port = cfg->port.local_udp_port;
+ const int ipv6 = (cfg->port.family == AF_INET6);
+ int err;
+
+ if (!sock)
+ err = udp_sock_create(net, &cfg->port, &sock);
+ else
+ err = (sock->sk->sk_protocol == IPPROTO_UDP) ?
+ 0 : -EPROTONOSUPPORT;
+
+ if (err)
+ return NULL;
+
+ uts = kzalloc(size, GFP_KERNEL);
+ if (!uts)
+ return ERR_PTR(-ENOMEM);
+
+ sk = sock->sk;
+
+ /* Disable multicast loopback */
+ inet_sk(sk)->mc_loop = 0;
+
+ rcu_assign_sk_user_data(sk, uts);
+
+ udp_sk(sk)->encap_type = cfg->encap_type;
+ udp_sk(sk)->encap_rcv = cfg->encap_rcv;
+ udp_sk(sk)->encap_destroy = cfg->encap_destroy;
+
+ uts->sock = sock;
+
+ spin_lock(&utn->sock_lock);
+ hlist_add_head_rcu(&uts->hlist, uts_head(net, port));
+ spin_unlock(&utn->sock_lock);
+
+#if IS_ENABLED(CONFIG_IPV6)
+ if (ipv6)
+ ipv6_stub->udpv6_encap_enable();
+ else
+#endif
+ udp_encap_enable();
+
+ return uts;
+}
+EXPORT_SYMBOL_GPL(create_udp_tunnel_sock);
+
+int udp_tunnel_xmit_skb(struct socket *sock, struct rtable *rt,
+ struct sk_buff *skb, __be32 src, __be32 dst,
+ __u8 tos, __u8 ttl, __be16 df, __be16 src_port,
+ __be16 dst_port, bool xnet)
+{
+ struct udphdr *uh;
+
+ __skb_push(skb, sizeof(*uh));
+ skb_reset_transport_header(skb);
+ uh = udp_hdr(skb);
+
+ uh->dest = dst_port;
+ uh->source = src_port;
+ uh->len = htons(skb->len);
+
+ udp_set_csum(sock->sk->sk_no_check_tx, skb, src, dst, skb->len);
+
+ return iptunnel_xmit(sock->sk, rt, skb, src, dst, IPPROTO_UDP,
+ tos, ttl, df, xnet);
+}
+EXPORT_SYMBOL_GPL(udp_tunnel_xmit_skb);
+
+#if IS_ENABLED(CONFIG_IPV6)
+int udp_tunnel6_xmit_skb(struct socket *sock, struct dst_entry *dst,
+ struct sk_buff *skb, struct net_device *dev,
+ struct in6_addr *saddr, struct in6_addr *daddr,
+ __u8 prio, __u8 ttl, __be16 src_port, __be16 dst_port)
+{
+ struct udphdr *uh;
+ struct ipv6hdr *ip6h;
+
+ __skb_push(skb, sizeof(*uh));
+ skb_reset_transport_header(skb);
+ uh = udp_hdr(skb);
+
+ uh->dest = dst_port;
+ uh->source = src_port;
+
+ uh->len = htons(skb->len);
+ uh->check = 0;
+
+ memset(&(IPCB(skb)->opt), 0, sizeof(IPCB(skb)->opt));
+ IPCB(skb)->flags &= ~(IPSKB_XFRM_TUNNEL_SIZE | IPSKB_XFRM_TRANSFORMED
+ | IPSKB_REROUTED);
+ skb_dst_set(skb, dst);
+
+ if (!skb_is_gso(skb) && !(dst->dev->features & NETIF_F_IPV6_CSUM)) {
+ __wsum csum = skb_checksum(skb, 0, skb->len, 0);
+
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+ uh->check = csum_ipv6_magic(saddr, daddr, skb->len,
+ IPPROTO_UDP, csum);
+ if (uh->check == 0)
+ uh->check = CSUM_MANGLED_0;
+ } else {
+ skb->ip_summed = CHECKSUM_PARTIAL;
+ skb->csum_start = skb_transport_header(skb) - skb->head;
+ skb->csum_offset = offsetof(struct udphdr, check);
+ uh->check = ~csum_ipv6_magic(saddr, daddr,
+ skb->len, IPPROTO_UDP, 0);
+ }
+
+ __skb_push(skb, sizeof(*ip6h));
+ skb_reset_network_header(skb);
+ ip6h = ipv6_hdr(skb);
+ ip6h->version = 6;
+ ip6h->priority = prio;
+ ip6h->flow_lbl[0] = 0;
+ ip6h->flow_lbl[1] = 0;
+ ip6h->flow_lbl[2] = 0;
+ ip6h->payload_len = htons(skb->len);
+ ip6h->nexthdr = IPPROTO_UDP;
+ ip6h->hop_limit = ttl;
+ ip6h->daddr = *daddr;
+ ip6h->saddr = *saddr;
+
+ ip6tunnel_xmit(skb, dev);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(udp_tunnel6_xmit_skb);
+#endif
+
+struct udp_tunnel_sock *udp_tunnel_find_sock(struct net *net, __be16 port)
+{
+ struct udp_tunnel_sock *uts;
+
+ hlist_for_each_entry_rcu(uts, uts_head(net, port), hlist) {
+ if (inet_sk(uts->sock->sk)->inet_sport == port)
+ return uts;
+ }
+
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(udp_tunnel_find_sock);
+
+void udp_tunnel_sock_release(struct udp_tunnel_sock *uts)
+{
+ struct sock *sk = uts->sock->sk;
+ struct net *net = sock_net(sk);
+ struct udp_tunnel_net *utn = net_generic(net, udp_tunnel_net_id);
+
+ spin_lock(&utn->sock_lock);
+ hlist_del_rcu(&uts->hlist);
+ rcu_assign_sk_user_data(uts->sock->sk, NULL);
+ kernel_sock_shutdown(uts->sock, SHUT_RDWR);
+ sk_release_kernel(sk);
+ spin_unlock(&utn->sock_lock);
+}
+EXPORT_SYMBOL_GPL(udp_tunnel_sock_release);
+
+static int __net_init udp_tunnel_init_net(struct net *net)
+{
+ struct udp_tunnel_net *utn = net_generic(net, udp_tunnel_net_id);
+ unsigned int h;
+
+ spin_lock_init(&utn->sock_lock);
+
+ for (h = 0; h < PORT_HASH_SIZE; h++)
+ INIT_HLIST_HEAD(&utn->sock_list[h]);
+
+ return 0;
+}
+
+static struct pernet_operations udp_tunnel_net_ops = {
+ .init = udp_tunnel_init_net,
+ .exit = NULL,
+ .id = &udp_tunnel_net_id,
+ .size = sizeof(struct udp_tunnel_net),
+};
+
+static int __init udp_tunnel_init(void)
+{
+ return register_pernet_subsys(&udp_tunnel_net_ops);
+}
+late_initcall(udp_tunnel_init);
+
MODULE_LICENSE("GPL");
--
1.7.9.5
^ permalink raw reply related
* [net-next v3 2/3] vxlan: Refactor vxlan driver to make use of the common UDP tunnel functions.
From: Andy Zhou @ 2014-08-27 3:35 UTC (permalink / raw)
To: davem; +Cc: netdev, Andy Zhou
In-Reply-To: <1409110530-4383-1-git-send-email-azhou@nicira.com>
Signed-off-by: Andy Zhou <azhou@nicira.com>
---
drivers/net/vxlan.c | 199 ++++++++++++-----------------------------
include/net/vxlan.h | 16 ++--
net/openvswitch/vport-vxlan.c | 6 +-
3 files changed, 68 insertions(+), 153 deletions(-)
diff --git a/drivers/net/vxlan.c b/drivers/net/vxlan.c
index beb377b..f1f1c48 100644
--- a/drivers/net/vxlan.c
+++ b/drivers/net/vxlan.c
@@ -42,6 +42,7 @@
#include <net/netns/generic.h>
#include <net/vxlan.h>
#include <net/protocol.h>
+#include <net/udp_tunnel.h>
#if IS_ENABLED(CONFIG_IPV6)
#include <net/ipv6.h>
#include <net/addrconf.h>
@@ -277,13 +278,7 @@ static inline struct vxlan_rdst *first_remote_rtnl(struct vxlan_fdb *fdb)
/* Find VXLAN socket based on network namespace and UDP port */
static struct vxlan_sock *vxlan_find_sock(struct net *net, __be16 port)
{
- struct vxlan_sock *vs;
-
- hlist_for_each_entry_rcu(vs, vs_head(net, port), hlist) {
- if (inet_sk(vs->sock->sk)->inet_sport == port)
- return vs;
- }
- return NULL;
+ return (struct vxlan_sock *)udp_tunnel_find_sock(net, port);
}
static struct vxlan_dev *vxlan_vs_find_vni(struct vxlan_sock *vs, u32 id)
@@ -636,7 +631,7 @@ static int vxlan_gro_complete(struct sk_buff *skb, int nhoff)
static void vxlan_notify_add_rx_port(struct vxlan_sock *vs)
{
struct net_device *dev;
- struct sock *sk = vs->sock->sk;
+ struct sock *sk = vs->uts.sock->sk;
struct net *net = sock_net(sk);
sa_family_t sa_family = sk->sk_family;
__be16 port = inet_sk(sk)->inet_sport;
@@ -661,7 +656,7 @@ static void vxlan_notify_add_rx_port(struct vxlan_sock *vs)
static void vxlan_notify_del_rx_port(struct vxlan_sock *vs)
{
struct net_device *dev;
- struct sock *sk = vs->sock->sk;
+ struct sock *sk = vs->uts.sock->sk;
struct net *net = sock_net(sk);
sa_family_t sa_family = sk->sk_family;
__be16 port = inet_sk(sk)->inet_sport;
@@ -1053,7 +1048,7 @@ static void vxlan_sock_hold(struct vxlan_sock *vs)
void vxlan_sock_release(struct vxlan_sock *vs)
{
- struct sock *sk = vs->sock->sk;
+ struct sock *sk = vs->uts.sock->sk;
struct net *net = sock_net(sk);
struct vxlan_net *vn = net_generic(net, vxlan_net_id);
@@ -1062,7 +1057,6 @@ void vxlan_sock_release(struct vxlan_sock *vs)
spin_lock(&vn->sock_lock);
hlist_del_rcu(&vs->hlist);
- rcu_assign_sk_user_data(vs->sock->sk, NULL);
vxlan_notify_del_rx_port(vs);
spin_unlock(&vn->sock_lock);
@@ -1078,7 +1072,7 @@ static void vxlan_igmp_join(struct work_struct *work)
{
struct vxlan_dev *vxlan = container_of(work, struct vxlan_dev, igmp_join);
struct vxlan_sock *vs = vxlan->vn_sock;
- struct sock *sk = vs->sock->sk;
+ struct sock *sk = vs->uts.sock->sk;
union vxlan_addr *ip = &vxlan->default_dst.remote_ip;
int ifindex = vxlan->default_dst.remote_ifindex;
@@ -1107,7 +1101,7 @@ static void vxlan_igmp_leave(struct work_struct *work)
{
struct vxlan_dev *vxlan = container_of(work, struct vxlan_dev, igmp_leave);
struct vxlan_sock *vs = vxlan->vn_sock;
- struct sock *sk = vs->sock->sk;
+ struct sock *sk = vs->uts.sock->sk;
union vxlan_addr *ip = &vxlan->default_dst.remote_ip;
int ifindex = vxlan->default_dst.remote_ifindex;
@@ -1338,7 +1332,6 @@ out:
}
#if IS_ENABLED(CONFIG_IPV6)
-
static struct sk_buff *vxlan_na_create(struct sk_buff *request,
struct neighbour *n, bool isrouter)
{
@@ -1572,13 +1565,6 @@ static bool route_shortcircuit(struct net_device *dev, struct sk_buff *skb)
return false;
}
-static inline struct sk_buff *vxlan_handle_offloads(struct sk_buff *skb,
- bool udp_csum)
-{
- int type = udp_csum ? SKB_GSO_UDP_TUNNEL_CSUM : SKB_GSO_UDP_TUNNEL;
- return iptunnel_handle_offloads(skb, udp_csum, type);
-}
-
#if IS_ENABLED(CONFIG_IPV6)
static int vxlan6_xmit_skb(struct vxlan_sock *vs,
struct dst_entry *dst, struct sk_buff *skb,
@@ -1587,13 +1573,13 @@ static int vxlan6_xmit_skb(struct vxlan_sock *vs,
__be16 src_port, __be16 dst_port, __be32 vni,
bool xnet)
{
- struct ipv6hdr *ip6h;
struct vxlanhdr *vxh;
- struct udphdr *uh;
int min_headroom;
int err;
- skb = vxlan_handle_offloads(skb, !udp_get_no_check6_tx(vs->sock->sk));
+ skb = udp_tunnel_handle_offloads(skb,
+ !udp_get_no_check6_tx(
+ vs->uts.sock->sk));
if (IS_ERR(skb))
return -EINVAL;
@@ -1621,38 +1607,8 @@ static int vxlan6_xmit_skb(struct vxlan_sock *vs,
vxh->vx_flags = htonl(VXLAN_FLAGS);
vxh->vx_vni = vni;
- __skb_push(skb, sizeof(*uh));
- skb_reset_transport_header(skb);
- uh = udp_hdr(skb);
-
- uh->dest = dst_port;
- uh->source = src_port;
-
- uh->len = htons(skb->len);
-
- memset(&(IPCB(skb)->opt), 0, sizeof(IPCB(skb)->opt));
- IPCB(skb)->flags &= ~(IPSKB_XFRM_TUNNEL_SIZE | IPSKB_XFRM_TRANSFORMED |
- IPSKB_REROUTED);
- skb_dst_set(skb, dst);
-
- udp6_set_csum(udp_get_no_check6_tx(vs->sock->sk), skb,
- saddr, daddr, skb->len);
-
- __skb_push(skb, sizeof(*ip6h));
- skb_reset_network_header(skb);
- ip6h = ipv6_hdr(skb);
- ip6h->version = 6;
- ip6h->priority = prio;
- ip6h->flow_lbl[0] = 0;
- ip6h->flow_lbl[1] = 0;
- ip6h->flow_lbl[2] = 0;
- ip6h->payload_len = htons(skb->len);
- ip6h->nexthdr = IPPROTO_UDP;
- ip6h->hop_limit = ttl;
- ip6h->daddr = *daddr;
- ip6h->saddr = *saddr;
-
- ip6tunnel_xmit(skb, dev);
+ udp_tunnel6_xmit_skb(vs->uts.sock, dst, skb, dev, saddr, daddr, prio,
+ ttl, src_port, dst_port);
return 0;
}
#endif
@@ -1663,11 +1619,11 @@ int vxlan_xmit_skb(struct vxlan_sock *vs,
__be16 src_port, __be16 dst_port, __be32 vni, bool xnet)
{
struct vxlanhdr *vxh;
- struct udphdr *uh;
int min_headroom;
int err;
- skb = vxlan_handle_offloads(skb, !vs->sock->sk->sk_no_check_tx);
+ skb = udp_tunnel_handle_offloads(skb,
+ !vs->uts.sock->sk->sk_no_check_tx);
if (IS_ERR(skb))
return -EINVAL;
@@ -1693,20 +1649,8 @@ int vxlan_xmit_skb(struct vxlan_sock *vs,
vxh->vx_flags = htonl(VXLAN_FLAGS);
vxh->vx_vni = vni;
- __skb_push(skb, sizeof(*uh));
- skb_reset_transport_header(skb);
- uh = udp_hdr(skb);
-
- uh->dest = dst_port;
- uh->source = src_port;
-
- uh->len = htons(skb->len);
-
- udp_set_csum(vs->sock->sk->sk_no_check_tx, skb,
- src, dst, skb->len);
-
- return iptunnel_xmit(vs->sock->sk, rt, skb, src, dst, IPPROTO_UDP,
- tos, ttl, df, xnet);
+ return udp_tunnel_xmit_skb(vs->uts.sock, rt, skb, src, dst, tos,
+ ttl, df, src_port, dst_port, xnet);
}
EXPORT_SYMBOL_GPL(vxlan_xmit_skb);
@@ -1831,18 +1775,18 @@ static void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev,
tos = ip_tunnel_ecn_encap(tos, old_iph, skb);
ttl = ttl ? : ip4_dst_hoplimit(&rt->dst);
- err = vxlan_xmit_skb(vxlan->vn_sock, rt, skb,
- fl4.saddr, dst->sin.sin_addr.s_addr,
- tos, ttl, df, src_port, dst_port,
- htonl(vni << 8),
- !net_eq(vxlan->net, dev_net(vxlan->dev)));
+ err = udp_tunnel_xmit_skb(vxlan->vn_sock->uts.sock, rt, skb,
+ fl4.saddr, dst->sin.sin_addr.s_addr,
+ tos, ttl, df, src_port, dst_port,
+ !net_eq(vxlan->net,
+ dev_net(vxlan->dev)));
if (err < 0)
goto rt_tx_error;
iptunnel_xmit_stats(err, &dev->stats, dev->tstats);
#if IS_ENABLED(CONFIG_IPV6)
} else {
- struct sock *sk = vxlan->vn_sock->sock->sk;
+ struct sock *sk = vxlan->vn_sock->uts.sock->sk;
struct dst_entry *ndst;
struct flowi6 fl6;
u32 flags;
@@ -2204,8 +2148,8 @@ void vxlan_get_rx_port(struct net_device *dev)
spin_lock(&vn->sock_lock);
for (i = 0; i < PORT_HASH_SIZE; ++i) {
hlist_for_each_entry_rcu(vs, &vn->sock_list[i], hlist) {
- port = inet_sk(vs->sock->sk)->inet_sport;
- sa_family = vs->sock->sk->sk_family;
+ port = inet_sk(vs->uts.sock->sk)->inet_sport;
+ sa_family = vs->uts.sock->sk->sk_family;
dev->netdev_ops->ndo_add_vxlan_port(dev, sa_family,
port);
}
@@ -2335,79 +2279,60 @@ static const struct ethtool_ops vxlan_ethtool_ops = {
static void vxlan_del_work(struct work_struct *work)
{
struct vxlan_sock *vs = container_of(work, struct vxlan_sock, del_work);
-
- sk_release_kernel(vs->sock->sk);
+ udp_tunnel_sock_release(&vs->uts);
kfree_rcu(vs, rcu);
}
-static struct socket *vxlan_create_sock(struct net *net, bool ipv6,
- __be16 port, u32 flags)
+/* Create new listen socket if needed */
+static struct vxlan_sock *vxlan_socket_create(struct net *net, __be16 port,
+ vxlan_rcv_t rcv, void *data,
+ u32 flags)
{
- struct socket *sock;
- struct udp_port_cfg udp_conf;
- int err;
+ struct vxlan_net *vn = net_generic(net, vxlan_net_id);
+ struct vxlan_sock *vs;
+ struct udp_tunnel_socket_cfg vxlan_ts_cfg;
+ bool ipv6 = !!(flags & VXLAN_F_IPV6);
+ unsigned int h;
- memset(&udp_conf, 0, sizeof(udp_conf));
+ memset(&vxlan_ts_cfg, 0, sizeof(struct udp_tunnel_socket_cfg));
if (ipv6) {
- udp_conf.family = AF_INET6;
- udp_conf.use_udp6_tx_checksums =
+ vxlan_ts_cfg.port.family = AF_INET6;
+ vxlan_ts_cfg.port.use_udp6_tx_checksums =
!!(flags & VXLAN_F_UDP_ZERO_CSUM6_TX);
- udp_conf.use_udp6_rx_checksums =
+ vxlan_ts_cfg.port.use_udp6_rx_checksums =
!!(flags & VXLAN_F_UDP_ZERO_CSUM6_RX);
} else {
- udp_conf.family = AF_INET;
- udp_conf.local_ip.s_addr = INADDR_ANY;
- udp_conf.use_udp_checksums =
+ vxlan_ts_cfg.port.family = AF_INET;
+ vxlan_ts_cfg.port.local_ip.s_addr = INADDR_ANY;
+ vxlan_ts_cfg.port.use_udp_checksums =
!!(flags & VXLAN_F_UDP_CSUM);
}
- udp_conf.local_udp_port = port;
+ vxlan_ts_cfg.port.local_udp_port = port;
+ vxlan_ts_cfg.encap_type = 1;
+ vxlan_ts_cfg.encap_rcv = vxlan_udp_encap_recv;
+ vxlan_ts_cfg.encap_destroy = NULL;
- /* Open UDP socket */
- err = udp_sock_create(net, &udp_conf, &sock);
- if (err < 0)
- return ERR_PTR(err);
-
- /* Disable multicast loopback */
- inet_sk(sock->sk)->mc_loop = 0;
-
- return sock;
-}
-
-/* Create new listen socket if needed */
-static struct vxlan_sock *vxlan_socket_create(struct net *net, __be16 port,
- vxlan_rcv_t *rcv, void *data,
- u32 flags)
-{
- struct vxlan_net *vn = net_generic(net, vxlan_net_id);
- struct vxlan_sock *vs;
- struct socket *sock;
- struct sock *sk;
- unsigned int h;
- bool ipv6 = !!(flags & VXLAN_F_IPV6);
-
- vs = kzalloc(sizeof(*vs), GFP_KERNEL);
+ vs = (struct vxlan_sock *)create_udp_tunnel_sock(net, sizeof(*vs),
+ NULL,
+ &vxlan_ts_cfg);
if (!vs)
return ERR_PTR(-ENOMEM);
for (h = 0; h < VNI_HASH_SIZE; ++h)
INIT_HLIST_HEAD(&vs->vni_list[h]);
- INIT_WORK(&vs->del_work, vxlan_del_work);
+ spin_lock(&vn->sock_lock);
+ list_add(&vs->next, &vn->vxlan_list);
+ spin_unlock(&vn->sock_lock);
- sock = vxlan_create_sock(net, ipv6, port, flags);
- if (IS_ERR(sock)) {
- kfree(vs);
- return ERR_CAST(sock);
- }
+ INIT_WORK(&vs->del_work, vxlan_del_work);
- vs->sock = sock;
- sk = sock->sk;
atomic_set(&vs->refcnt, 1);
+
vs->rcv = rcv;
- vs->data = data;
- rcu_assign_sk_user_data(vs->sock->sk, vs);
+ vs->rcv_data = data;
/* Initialize the vxlan udp offloads structure */
vs->udp_offloads.port = port;
@@ -2419,24 +2344,13 @@ static struct vxlan_sock *vxlan_socket_create(struct net *net, __be16 port,
vxlan_notify_add_rx_port(vs);
spin_unlock(&vn->sock_lock);
- /* Mark socket as an encapsulation socket. */
- udp_sk(sk)->encap_type = 1;
- udp_sk(sk)->encap_rcv = vxlan_udp_encap_recv;
-#if IS_ENABLED(CONFIG_IPV6)
- if (ipv6)
- ipv6_stub->udpv6_encap_enable();
- else
-#endif
- udp_encap_enable();
-
return vs;
}
struct vxlan_sock *vxlan_sock_add(struct net *net, __be16 port,
- vxlan_rcv_t *rcv, void *data,
+ vxlan_rcv_t rcv, void *data,
bool no_share, u32 flags)
{
- struct vxlan_net *vn = net_generic(net, vxlan_net_id);
struct vxlan_sock *vs;
vs = vxlan_socket_create(net, port, rcv, data, flags);
@@ -2446,7 +2360,6 @@ struct vxlan_sock *vxlan_sock_add(struct net *net, __be16 port,
if (no_share) /* Return error if sharing is not allowed. */
return vs;
- spin_lock(&vn->sock_lock);
vs = vxlan_find_sock(net, port);
if (vs) {
if (vs->rcv == rcv)
@@ -2454,7 +2367,6 @@ struct vxlan_sock *vxlan_sock_add(struct net *net, __be16 port,
else
vs = ERR_PTR(-EBUSY);
}
- spin_unlock(&vn->sock_lock);
if (!vs)
vs = ERR_PTR(-EINVAL);
@@ -2634,7 +2546,6 @@ static int vxlan_newlink(struct net *net, struct net_device *dev,
}
list_add(&vxlan->next, &vn->vxlan_list);
-
return 0;
}
diff --git a/include/net/vxlan.h b/include/net/vxlan.h
index d5f59f3..10bfc13 100644
--- a/include/net/vxlan.h
+++ b/include/net/vxlan.h
@@ -4,23 +4,27 @@
#include <linux/skbuff.h>
#include <linux/netdevice.h>
#include <linux/udp.h>
+#include <net/udp_tunnel.h>
#define VNI_HASH_BITS 10
#define VNI_HASH_SIZE (1<<VNI_HASH_BITS)
struct vxlan_sock;
-typedef void (vxlan_rcv_t)(struct vxlan_sock *vh, struct sk_buff *skb, __be32 key);
-/* per UDP socket information */
+typedef void (*vxlan_rcv_t)(struct vxlan_sock *vs, struct sk_buff *skb,
+ __be32 key);
+
+/* per vxlan socket information */
struct vxlan_sock {
+ struct udp_tunnel_sock uts; /* Must be the first member */
struct hlist_node hlist;
- vxlan_rcv_t *rcv;
- void *data;
+ struct list_head next;
struct work_struct del_work;
- struct socket *sock;
struct rcu_head rcu;
struct hlist_head vni_list[VNI_HASH_SIZE];
atomic_t refcnt;
+ vxlan_rcv_t rcv;
+ void *rcv_data;
struct udp_offload udp_offloads;
};
@@ -35,7 +39,7 @@ struct vxlan_sock {
#define VXLAN_F_UDP_ZERO_CSUM6_RX 0x100
struct vxlan_sock *vxlan_sock_add(struct net *net, __be16 port,
- vxlan_rcv_t *rcv, void *data,
+ vxlan_rcv_t rcv, void *data,
bool no_share, u32 flags);
void vxlan_sock_release(struct vxlan_sock *vs);
diff --git a/net/openvswitch/vport-vxlan.c b/net/openvswitch/vport-vxlan.c
index d8b7e24..7599efd 100644
--- a/net/openvswitch/vport-vxlan.c
+++ b/net/openvswitch/vport-vxlan.c
@@ -59,7 +59,7 @@ static inline struct vxlan_port *vxlan_vport(const struct vport *vport)
static void vxlan_rcv(struct vxlan_sock *vs, struct sk_buff *skb, __be32 vx_vni)
{
struct ovs_key_ipv4_tunnel tun_key;
- struct vport *vport = vs->data;
+ struct vport *vport = vs->rcv_data;
struct iphdr *iph;
__be64 key;
@@ -74,7 +74,7 @@ static void vxlan_rcv(struct vxlan_sock *vs, struct sk_buff *skb, __be32 vx_vni)
static int vxlan_get_options(const struct vport *vport, struct sk_buff *skb)
{
struct vxlan_port *vxlan_port = vxlan_vport(vport);
- __be16 dst_port = inet_sk(vxlan_port->vs->sock->sk)->inet_sport;
+ __be16 dst_port = inet_sk(vxlan_port->vs->uts.sock->sk)->inet_sport;
if (nla_put_u16(skb, OVS_TUNNEL_ATTR_DST_PORT, ntohs(dst_port)))
return -EMSGSIZE;
@@ -139,7 +139,7 @@ static int vxlan_tnl_send(struct vport *vport, struct sk_buff *skb)
{
struct net *net = ovs_dp_get_net(vport->dp);
struct vxlan_port *vxlan_port = vxlan_vport(vport);
- __be16 dst_port = inet_sk(vxlan_port->vs->sock->sk)->inet_sport;
+ __be16 dst_port = inet_sk(vxlan_port->vs->uts.sock->sk)->inet_sport;
struct rtable *rt;
struct flowi4 fl;
__be16 src_port;
--
1.7.9.5
^ permalink raw reply related
* [net-next v3 3/3] l2tp: Refactor l2tp core driver to make use of the common UDP tunnel functions
From: Andy Zhou @ 2014-08-27 3:35 UTC (permalink / raw)
To: davem; +Cc: netdev, Andy Zhou
In-Reply-To: <1409110530-4383-1-git-send-email-azhou@nicira.com>
Signed-off-by: Andy Zhou <azhou@nicira.com>
---
net/l2tp/l2tp_core.c | 189 +++++++++++++++++++++++++++++---------------------
1 file changed, 109 insertions(+), 80 deletions(-)
diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
index 1109d3b..b4b9474 100644
--- a/net/l2tp/l2tp_core.c
+++ b/net/l2tp/l2tp_core.c
@@ -113,6 +113,11 @@ struct l2tp_net {
spinlock_t l2tp_session_hlist_lock;
};
+struct l2tp_udp_tunnel {
+ struct udp_tunnel_sock uts;
+ struct l2tp_tunnel tunnel;
+};
+
static void l2tp_tunnel_free(struct l2tp_tunnel *tunnel);
static inline struct l2tp_tunnel *l2tp_tunnel(struct sock *sk)
@@ -1334,14 +1339,47 @@ static void l2tp_tunnel_del_work(struct work_struct *work)
if (sock)
inet_shutdown(sock, 2);
} else {
+ struct l2tp_udp_tunnel *udp_tunnel;
+
+ udp_tunnel = container_of(tunnel,
+ struct l2tp_udp_tunnel, tunnel);
+
if (sock)
- kernel_sock_shutdown(sock, SHUT_RDWR);
- sk_release_kernel(sk);
+ udp_tunnel_sock_release(&udp_tunnel->uts);
}
l2tp_tunnel_sock_put(sk);
}
+static void l2tp_tunnel_udp_port_conf(struct l2tp_tunnel_cfg *cfg,
+ struct udp_port_cfg *udp_conf)
+{
+ memset(udp_conf, 0, sizeof(*udp_conf));
+
+#if IS_ENABLED(CONFIG_IPV6)
+ if (cfg->local_ip6 && cfg->peer_ip6) {
+ udp_conf->family = AF_INET6;
+ memcpy(&udp_conf->local_ip6, cfg->local_ip6,
+ sizeof(udp_conf->local_ip6));
+ memcpy(&udp_conf->peer_ip6, cfg->peer_ip6,
+ sizeof(udp_conf->peer_ip6));
+ udp_conf->use_udp6_tx_checksums =
+ cfg->udp6_zero_tx_checksums;
+ udp_conf->use_udp6_rx_checksums =
+ cfg->udp6_zero_rx_checksums;
+ } else
+#endif
+ {
+ udp_conf->family = AF_INET;
+ udp_conf->local_ip = cfg->local_ip;
+ udp_conf->peer_ip = cfg->peer_ip;
+ udp_conf->use_udp_checksums = cfg->use_udp_checksums;
+ }
+
+ udp_conf->local_udp_port = htons(cfg->local_udp_port);
+ udp_conf->peer_udp_port = htons(cfg->peer_udp_port);
+}
+
/* Create a socket for the tunnel, if one isn't set up by
* userspace. This is used for static tunnels where there is no
* managing L2TP daemon.
@@ -1363,31 +1401,7 @@ static int l2tp_tunnel_sock_create(struct net *net,
switch (cfg->encap) {
case L2TP_ENCAPTYPE_UDP:
- memset(&udp_conf, 0, sizeof(udp_conf));
-
-#if IS_ENABLED(CONFIG_IPV6)
- if (cfg->local_ip6 && cfg->peer_ip6) {
- udp_conf.family = AF_INET6;
- memcpy(&udp_conf.local_ip6, cfg->local_ip6,
- sizeof(udp_conf.local_ip6));
- memcpy(&udp_conf.peer_ip6, cfg->peer_ip6,
- sizeof(udp_conf.peer_ip6));
- udp_conf.use_udp6_tx_checksums =
- cfg->udp6_zero_tx_checksums;
- udp_conf.use_udp6_rx_checksums =
- cfg->udp6_zero_rx_checksums;
- } else
-#endif
- {
- udp_conf.family = AF_INET;
- udp_conf.local_ip = cfg->local_ip;
- udp_conf.peer_ip = cfg->peer_ip;
- udp_conf.use_udp_checksums = cfg->use_udp_checksums;
- }
-
- udp_conf.local_udp_port = htons(cfg->local_udp_port);
- udp_conf.peer_udp_port = htons(cfg->peer_udp_port);
-
+ l2tp_tunnel_udp_port_conf(cfg, &udp_conf);
err = udp_sock_create(net, &udp_conf, &sock);
if (err < 0)
goto out;
@@ -1471,6 +1485,31 @@ out:
static struct lock_class_key l2tp_socket_class;
+static int l2tp_sk_sanity_check(struct sock *sk, enum l2tp_encap_type encap,
+ u32 tunnel_id, int fd)
+{
+ unsigned int expected_protocol;
+
+ switch (encap) {
+ case L2TP_ENCAPTYPE_UDP:
+ expected_protocol = IPPROTO_UDP;
+ break;
+ case L2TP_ENCAPTYPE_IP:
+ expected_protocol = IPPROTO_L2TP;
+ break;
+ default:
+ return -EPROTONOSUPPORT;
+ }
+
+ if (sk->sk_protocol != expected_protocol) {
+ pr_err("tunl %hu: fd %d wrong protocol, got %d, expected %d\n",
+ tunnel_id, fd, sk->sk_protocol, expected_protocol);
+ return -EPROTONOSUPPORT;
+ }
+
+ return 0;
+}
+
int l2tp_tunnel_create(struct net *net, int fd, int version, u32 tunnel_id, u32 peer_tunnel_id, struct l2tp_tunnel_cfg *cfg, struct l2tp_tunnel **tunnelp)
{
struct l2tp_tunnel *tunnel = NULL;
@@ -1478,7 +1517,7 @@ int l2tp_tunnel_create(struct net *net, int fd, int version, u32 tunnel_id, u32
struct socket *sock = NULL;
struct sock *sk = NULL;
struct l2tp_net *pn;
- enum l2tp_encap_type encap = L2TP_ENCAPTYPE_UDP;
+ enum l2tp_encap_type encap = cfg ? cfg->encap : L2TP_ENCAPTYPE_UDP;
/* Get the tunnel socket from the fd, which was opened by
* the userspace L2TP daemon. If not specified, create a
@@ -1486,9 +1525,11 @@ int l2tp_tunnel_create(struct net *net, int fd, int version, u32 tunnel_id, u32
*/
if (fd < 0) {
err = l2tp_tunnel_sock_create(net, tunnel_id, peer_tunnel_id,
- cfg, &sock);
+ cfg, &sock);
if (err < 0)
goto err;
+
+ sk = sock->sk;
} else {
sock = sockfd_lookup(fd, &err);
if (!sock) {
@@ -1498,58 +1539,66 @@ int l2tp_tunnel_create(struct net *net, int fd, int version, u32 tunnel_id, u32
goto err;
}
+ sk = sock->sk;
+
/* Reject namespace mismatches */
- if (!net_eq(sock_net(sock->sk), net)) {
+ if (!net_eq(sock_net(sk), net)) {
pr_err("tunl %u: netns mismatch\n", tunnel_id);
err = -EINVAL;
goto err;
}
+
+ /* Quick sanity checks */
+ err = l2tp_sk_sanity_check(sk, encap, tunnel_id, fd);
+ if (err)
+ goto err;
+
+ /* Check if this socket has already been prepped */
+ tunnel = l2tp_tunnel(sk);
+ if (tunnel != NULL) {
+ /* This socket has already been prepped */
+ err = -EBUSY;
+ goto err;
+ }
}
- sk = sock->sk;
+ switch(encap) {
+ case L2TP_ENCAPTYPE_UDP: {
+ struct udp_tunnel_socket_cfg l2tp_udp_cfg;
+ struct l2tp_udp_tunnel *udp_tunnel;
- if (cfg != NULL)
- encap = cfg->encap;
+ l2tp_tunnel_udp_port_conf(cfg, &l2tp_udp_cfg.port);
+ l2tp_udp_cfg.encap_type = UDP_ENCAP_L2TPINUDP;
+ l2tp_udp_cfg.encap_rcv = l2tp_udp_encap_recv;
+ l2tp_udp_cfg.encap_destroy = l2tp_udp_encap_destroy;
- /* Quick sanity checks */
- switch (encap) {
- case L2TP_ENCAPTYPE_UDP:
- err = -EPROTONOSUPPORT;
- if (sk->sk_protocol != IPPROTO_UDP) {
- pr_err("tunl %hu: fd %d wrong protocol, got %d, expected %d\n",
- tunnel_id, fd, sk->sk_protocol, IPPROTO_UDP);
+ udp_tunnel = (struct l2tp_udp_tunnel *)
+ create_udp_tunnel_sock(net, sizeof(*udp_tunnel),
+ sock, &l2tp_udp_cfg);
+
+ if (!udp_tunnel) {
+ err = -ENOMEM;
goto err;
}
+
+ tunnel = &udp_tunnel->tunnel;
break;
+ }
case L2TP_ENCAPTYPE_IP:
- err = -EPROTONOSUPPORT;
- if (sk->sk_protocol != IPPROTO_L2TP) {
- pr_err("tunl %hu: fd %d wrong protocol, got %d, expected %d\n",
- tunnel_id, fd, sk->sk_protocol, IPPROTO_L2TP);
+ tunnel = kzalloc(sizeof(*tunnel), GFP_KERNEL);
+ if (tunnel == NULL) {
+ err = -ENOMEM;
goto err;
}
- break;
- }
-
- /* Check if this socket has already been prepped */
- tunnel = l2tp_tunnel(sk);
- if (tunnel != NULL) {
- /* This socket has already been prepped */
- err = -EBUSY;
- goto err;
}
- tunnel = kzalloc(sizeof(struct l2tp_tunnel), GFP_KERNEL);
- if (tunnel == NULL) {
- err = -ENOMEM;
- goto err;
- }
+ rcu_assign_sk_user_data(sk, tunnel);
+ tunnel->encap = encap;
tunnel->version = version;
tunnel->tunnel_id = tunnel_id;
tunnel->peer_tunnel_id = peer_tunnel_id;
- tunnel->debug = L2TP_DEFAULT_DEBUG_FLAGS;
-
+ tunnel->debug = cfg ? cfg->debug : L2TP_DEFAULT_DEBUG_FLAGS;
tunnel->magic = L2TP_TUNNEL_MAGIC;
sprintf(&tunnel->name[0], "tunl %u", tunnel_id);
rwlock_init(&tunnel->hlist_lock);
@@ -1558,9 +1607,6 @@ int l2tp_tunnel_create(struct net *net, int fd, int version, u32 tunnel_id, u32
tunnel->l2tp_net = net;
pn = l2tp_pernet(net);
- if (cfg != NULL)
- tunnel->debug = cfg->debug;
-
#if IS_ENABLED(CONFIG_IPV6)
if (sk->sk_family == PF_INET6) {
struct ipv6_pinfo *np = inet6_sk(sk);
@@ -1579,23 +1625,6 @@ int l2tp_tunnel_create(struct net *net, int fd, int version, u32 tunnel_id, u32
}
#endif
- /* Mark socket as an encapsulation socket. See net/ipv4/udp.c */
- tunnel->encap = encap;
- if (encap == L2TP_ENCAPTYPE_UDP) {
- /* Mark socket as an encapsulation socket. See net/ipv4/udp.c */
- udp_sk(sk)->encap_type = UDP_ENCAP_L2TPINUDP;
- udp_sk(sk)->encap_rcv = l2tp_udp_encap_recv;
- udp_sk(sk)->encap_destroy = l2tp_udp_encap_destroy;
-#if IS_ENABLED(CONFIG_IPV6)
- if (sk->sk_family == PF_INET6 && !tunnel->v4mapped)
- udpv6_encap_enable();
- else
-#endif
- udp_encap_enable();
- }
-
- sk->sk_user_data = tunnel;
-
/* Hook on the tunnel socket destructor so that we can cleanup
* if the tunnel socket goes away.
*/
--
1.7.9.5
^ permalink raw reply related
* Re: [net-next v3 1/3] udp: Expand UDP tunnel common APIs
From: Eric Dumazet @ 2014-08-27 3:54 UTC (permalink / raw)
To: Andy Zhou; +Cc: davem, netdev
In-Reply-To: <1409110530-4383-1-git-send-email-azhou@nicira.com>
On Tue, 2014-08-26 at 20:35 -0700, Andy Zhou wrote:
> Added create_udp_tunnel_socket(), packet receive and transmit, and
> other related common functions for UDP tunnels.
>
> Per net open UDP tunnel ports are tracked in this common layer to
> prevent sharing of a single port with more than one UDP tunnel.
>
> Signed-off-by: Andy Zhou <azhou@nicira.com>
> ---
I see some RCU code, but I do not see elementary rules of RCU being
respected in this patch.
Changelogs are a bit terse, and there is not a single comment in this
code.
What is going on exactly ?
^ permalink raw reply
* Re: [PATCH RFC v7 net-next 00/28] BPF syscall
From: Andy Lutomirski @ 2014-08-27 3:56 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Daniel Borkmann, H. Peter Anvin, Andrew Morton, Chema Gonzalez,
Namhyung Kim, Eric Dumazet, David S. Miller,
linux-kernel@vger.kernel.org, Brendan Gregg, Linus Torvalds,
Steven Rostedt, Network Development, Peter Zijlstra, Kees Cook,
Linux API, Ingo Molnar
In-Reply-To: <1409106582-10095-1-git-send-email-ast@plumgrid.com>
On Aug 26, 2014 7:29 PM, "Alexei Starovoitov" <ast@plumgrid.com> wrote:
>
> Hi Ingo, David,
>
> posting whole thing again as RFC to get feedback on syscall only.
> If syscall bpf(int cmd, union bpf_attr *attr, unsigned int size) is ok,
> I'll split them into small chunks as requested and will repost without RFC.
IMO it's much easier to review a syscall if we just look at a
specification of what it does. The code is, in some sense, secondary.
--Andy
^ permalink raw reply
* Re: [net-next v3 1/3] udp: Expand UDP tunnel common APIs
From: Tom Herbert @ 2014-08-27 4:12 UTC (permalink / raw)
To: Andy Zhou; +Cc: David Miller, Linux Netdev List
In-Reply-To: <1409110530-4383-1-git-send-email-azhou@nicira.com>
On Tue, Aug 26, 2014 at 8:35 PM, Andy Zhou <azhou@nicira.com> wrote:
> Added create_udp_tunnel_socket(), packet receive and transmit, and
> other related common functions for UDP tunnels.
>
> Per net open UDP tunnel ports are tracked in this common layer to
> prevent sharing of a single port with more than one UDP tunnel.
>
This is not needed! If a UDP port is already bound (whether by another
tunnel or not), then bind during tunnel initialization will fail. All
this logic to store tunnel sockets in a separate list seems like
unnecessary complexity. If a driver needs to track multiple ports it
opens, it can do that on its own like VXLAN is already doing.
> Signed-off-by: Andy Zhou <azhou@nicira.com>
> ---
> include/net/udp_tunnel.h | 50 +++++++++++
> net/ipv4/udp_tunnel.c | 216 +++++++++++++++++++++++++++++++++++++++++++++-
> 2 files changed, 265 insertions(+), 1 deletion(-)
>
> diff --git a/include/net/udp_tunnel.h b/include/net/udp_tunnel.h
> index ffd69cb..5ff0746 100644
> --- a/include/net/udp_tunnel.h
> +++ b/include/net/udp_tunnel.h
> @@ -1,6 +1,8 @@
> #ifndef __NET_UDP_TUNNEL_H
> #define __NET_UDP_TUNNEL_H
>
> +#include <net/ip_tunnels.h>
> +
> struct udp_port_cfg {
> u8 family;
>
> @@ -26,7 +28,55 @@ struct udp_port_cfg {
> use_udp6_rx_checksums:1;
> };
>
> +struct udp_tunnel_sock;
> +
> +typedef int (*udp_tunnel_encap_rcv_t)(struct sock *sk, struct sk_buff *skb);
> +typedef void (*udp_tunnel_encap_destroy_t)(struct sock *sk);
> +
> +struct udp_tunnel_socket_cfg {
> + struct udp_port_cfg port;
> +
> + /* Used for setting up udp_sock fields, see udp.h for details */
> + __u8 encap_type;
> + udp_tunnel_encap_rcv_t encap_rcv;
> + udp_tunnel_encap_destroy_t encap_destroy;
> +};
> +
> +struct udp_tunnel_sock {
> + struct hlist_node hlist;
> + struct socket *sock;
> +};
> +
> int udp_sock_create(struct net *net, struct udp_port_cfg *cfg,
> struct socket **sockp);
>
> +struct udp_tunnel_sock *create_udp_tunnel_sock(struct net *net, size_t size,
> + struct socket *sock,
> + struct udp_tunnel_socket_cfg
> + *socket_cfg);
> +
> +struct udp_tunnel_sock *udp_tunnel_find_sock(struct net *net, __be16 port);
> +
> +int udp_tunnel_xmit_skb(struct socket *sock, struct rtable *rt,
> + struct sk_buff *skb, __be32 src, __be32 dst,
> + __u8 tos, __u8 ttl, __be16 df, __be16 src_port,
> + __be16 dst_port, bool xnet);
> +
> +#if IS_ENABLED(CONFIG_IPV6)
> +int udp_tunnel6_xmit_skb(struct socket *sock, struct dst_entry *dst,
> + struct sk_buff *skb, struct net_device *dev,
> + struct in6_addr *saddr, struct in6_addr *daddr,
> + __u8 prio, __u8 ttl, __be16 src_port, __be16 dst_port);
> +
> +#endif
> +
> +void udp_tunnel_sock_release(struct udp_tunnel_sock *uts);
> +
> +static inline struct sk_buff *udp_tunnel_handle_offloads(struct sk_buff *skb,
> + bool udp_csum)
> +{
> + int type = udp_csum ? SKB_GSO_UDP_TUNNEL_CSUM : SKB_GSO_UDP_TUNNEL;
> +
> + return iptunnel_handle_offloads(skb, udp_csum, type);
> +}
> #endif
> diff --git a/net/ipv4/udp_tunnel.c b/net/ipv4/udp_tunnel.c
> index 61ec1a6..3ed66bc 100644
> --- a/net/ipv4/udp_tunnel.c
> +++ b/net/ipv4/udp_tunnel.c
> @@ -7,6 +7,23 @@
> #include <net/udp.h>
> #include <net/udp_tunnel.h>
> #include <net/net_namespace.h>
> +#include <net/netns/generic.h>
> +#if IS_ENABLED(CONFIG_IPV6)
> +#include <net/ipv6.h>
> +#include <net/addrconf.h>
> +#include <net/ip6_tunnel.h>
> +#include <net/ip6_checksum.h>
> +#endif
> +
> +#define PORT_HASH_BITS 8
> +#define PORT_HASH_SIZE (1 << PORT_HASH_BITS)
> +
> +static int udp_tunnel_net_id;
> +
> +struct udp_tunnel_net {
> + struct hlist_head sock_list[PORT_HASH_SIZE];
> + spinlock_t sock_lock; /* Protecting the sock_list */
> +};
>
> int udp_sock_create(struct net *net, struct udp_port_cfg *cfg,
> struct socket **sockp)
> @@ -82,7 +99,6 @@ int udp_sock_create(struct net *net, struct udp_port_cfg *cfg,
> return -EPFNOSUPPORT;
> }
>
> -
> *sockp = sock;
>
> return 0;
> @@ -97,4 +113,202 @@ error:
> }
> EXPORT_SYMBOL(udp_sock_create);
>
> +
> +/* Socket hash table head */
> +static inline struct hlist_head *uts_head(struct net *net, const __be16 port)
> +{
> + struct udp_tunnel_net *utn = net_generic(net, udp_tunnel_net_id);
> +
> + return &utn->sock_list[hash_32(ntohs(port), PORT_HASH_BITS)];
> +}
> +
> +struct udp_tunnel_sock *create_udp_tunnel_sock(struct net *net, size_t size,
> + struct socket *sock,
> + struct udp_tunnel_socket_cfg
> + *cfg)
> +{
> + struct udp_tunnel_net *utn = net_generic(net, udp_tunnel_net_id);
> + struct udp_tunnel_sock *uts;
> + struct sock *sk;
> + const __be16 port = cfg->port.local_udp_port;
> + const int ipv6 = (cfg->port.family == AF_INET6);
> + int err;
> +
> + if (!sock)
> + err = udp_sock_create(net, &cfg->port, &sock);
> + else
> + err = (sock->sk->sk_protocol == IPPROTO_UDP) ?
> + 0 : -EPROTONOSUPPORT;
> +
> + if (err)
> + return NULL;
> +
> + uts = kzalloc(size, GFP_KERNEL);
> + if (!uts)
> + return ERR_PTR(-ENOMEM);
> +
> + sk = sock->sk;
> +
> + /* Disable multicast loopback */
> + inet_sk(sk)->mc_loop = 0;
> +
> + rcu_assign_sk_user_data(sk, uts);
> +
> + udp_sk(sk)->encap_type = cfg->encap_type;
> + udp_sk(sk)->encap_rcv = cfg->encap_rcv;
> + udp_sk(sk)->encap_destroy = cfg->encap_destroy;
> +
> + uts->sock = sock;
> +
> + spin_lock(&utn->sock_lock);
> + hlist_add_head_rcu(&uts->hlist, uts_head(net, port));
> + spin_unlock(&utn->sock_lock);
> +
> +#if IS_ENABLED(CONFIG_IPV6)
> + if (ipv6)
> + ipv6_stub->udpv6_encap_enable();
> + else
> +#endif
> + udp_encap_enable();
> +
> + return uts;
> +}
> +EXPORT_SYMBOL_GPL(create_udp_tunnel_sock);
> +
> +int udp_tunnel_xmit_skb(struct socket *sock, struct rtable *rt,
> + struct sk_buff *skb, __be32 src, __be32 dst,
> + __u8 tos, __u8 ttl, __be16 df, __be16 src_port,
> + __be16 dst_port, bool xnet)
> +{
> + struct udphdr *uh;
> +
> + __skb_push(skb, sizeof(*uh));
> + skb_reset_transport_header(skb);
> + uh = udp_hdr(skb);
> +
> + uh->dest = dst_port;
> + uh->source = src_port;
> + uh->len = htons(skb->len);
> +
> + udp_set_csum(sock->sk->sk_no_check_tx, skb, src, dst, skb->len);
> +
> + return iptunnel_xmit(sock->sk, rt, skb, src, dst, IPPROTO_UDP,
> + tos, ttl, df, xnet);
> +}
> +EXPORT_SYMBOL_GPL(udp_tunnel_xmit_skb);
> +
> +#if IS_ENABLED(CONFIG_IPV6)
> +int udp_tunnel6_xmit_skb(struct socket *sock, struct dst_entry *dst,
> + struct sk_buff *skb, struct net_device *dev,
> + struct in6_addr *saddr, struct in6_addr *daddr,
> + __u8 prio, __u8 ttl, __be16 src_port, __be16 dst_port)
> +{
> + struct udphdr *uh;
> + struct ipv6hdr *ip6h;
> +
> + __skb_push(skb, sizeof(*uh));
> + skb_reset_transport_header(skb);
> + uh = udp_hdr(skb);
> +
> + uh->dest = dst_port;
> + uh->source = src_port;
> +
> + uh->len = htons(skb->len);
> + uh->check = 0;
> +
> + memset(&(IPCB(skb)->opt), 0, sizeof(IPCB(skb)->opt));
> + IPCB(skb)->flags &= ~(IPSKB_XFRM_TUNNEL_SIZE | IPSKB_XFRM_TRANSFORMED
> + | IPSKB_REROUTED);
> + skb_dst_set(skb, dst);
> +
> + if (!skb_is_gso(skb) && !(dst->dev->features & NETIF_F_IPV6_CSUM)) {
> + __wsum csum = skb_checksum(skb, 0, skb->len, 0);
> +
> + skb->ip_summed = CHECKSUM_UNNECESSARY;
> + uh->check = csum_ipv6_magic(saddr, daddr, skb->len,
> + IPPROTO_UDP, csum);
> + if (uh->check == 0)
> + uh->check = CSUM_MANGLED_0;
> + } else {
> + skb->ip_summed = CHECKSUM_PARTIAL;
> + skb->csum_start = skb_transport_header(skb) - skb->head;
> + skb->csum_offset = offsetof(struct udphdr, check);
> + uh->check = ~csum_ipv6_magic(saddr, daddr,
> + skb->len, IPPROTO_UDP, 0);
> + }
> +
> + __skb_push(skb, sizeof(*ip6h));
> + skb_reset_network_header(skb);
> + ip6h = ipv6_hdr(skb);
> + ip6h->version = 6;
> + ip6h->priority = prio;
> + ip6h->flow_lbl[0] = 0;
> + ip6h->flow_lbl[1] = 0;
> + ip6h->flow_lbl[2] = 0;
> + ip6h->payload_len = htons(skb->len);
> + ip6h->nexthdr = IPPROTO_UDP;
> + ip6h->hop_limit = ttl;
> + ip6h->daddr = *daddr;
> + ip6h->saddr = *saddr;
> +
> + ip6tunnel_xmit(skb, dev);
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(udp_tunnel6_xmit_skb);
> +#endif
> +
> +struct udp_tunnel_sock *udp_tunnel_find_sock(struct net *net, __be16 port)
> +{
> + struct udp_tunnel_sock *uts;
> +
> + hlist_for_each_entry_rcu(uts, uts_head(net, port), hlist) {
> + if (inet_sk(uts->sock->sk)->inet_sport == port)
> + return uts;
> + }
> +
> + return NULL;
> +}
> +EXPORT_SYMBOL_GPL(udp_tunnel_find_sock);
> +
> +void udp_tunnel_sock_release(struct udp_tunnel_sock *uts)
> +{
> + struct sock *sk = uts->sock->sk;
> + struct net *net = sock_net(sk);
> + struct udp_tunnel_net *utn = net_generic(net, udp_tunnel_net_id);
> +
> + spin_lock(&utn->sock_lock);
> + hlist_del_rcu(&uts->hlist);
> + rcu_assign_sk_user_data(uts->sock->sk, NULL);
> + kernel_sock_shutdown(uts->sock, SHUT_RDWR);
> + sk_release_kernel(sk);
> + spin_unlock(&utn->sock_lock);
> +}
> +EXPORT_SYMBOL_GPL(udp_tunnel_sock_release);
> +
> +static int __net_init udp_tunnel_init_net(struct net *net)
> +{
> + struct udp_tunnel_net *utn = net_generic(net, udp_tunnel_net_id);
> + unsigned int h;
> +
> + spin_lock_init(&utn->sock_lock);
> +
> + for (h = 0; h < PORT_HASH_SIZE; h++)
> + INIT_HLIST_HEAD(&utn->sock_list[h]);
> +
> + return 0;
> +}
> +
> +static struct pernet_operations udp_tunnel_net_ops = {
> + .init = udp_tunnel_init_net,
> + .exit = NULL,
> + .id = &udp_tunnel_net_id,
> + .size = sizeof(struct udp_tunnel_net),
> +};
> +
> +static int __init udp_tunnel_init(void)
> +{
> + return register_pernet_subsys(&udp_tunnel_net_ops);
> +}
> +late_initcall(udp_tunnel_init);
> +
> MODULE_LICENSE("GPL");
> --
> 1.7.9.5
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [net-next v3 2/3] vxlan: Refactor vxlan driver to make use of the common UDP tunnel functions.
From: Tom Herbert @ 2014-08-27 4:15 UTC (permalink / raw)
To: Andy Zhou; +Cc: David Miller, Linux Netdev List
In-Reply-To: <1409110530-4383-2-git-send-email-azhou@nicira.com>
On Tue, Aug 26, 2014 at 8:35 PM, Andy Zhou <azhou@nicira.com> wrote:
> Signed-off-by: Andy Zhou <azhou@nicira.com>
> ---
> drivers/net/vxlan.c | 199 ++++++++++++-----------------------------
> include/net/vxlan.h | 16 ++--
> net/openvswitch/vport-vxlan.c | 6 +-
> 3 files changed, 68 insertions(+), 153 deletions(-)
>
> diff --git a/drivers/net/vxlan.c b/drivers/net/vxlan.c
> index beb377b..f1f1c48 100644
> --- a/drivers/net/vxlan.c
> +++ b/drivers/net/vxlan.c
> @@ -42,6 +42,7 @@
> #include <net/netns/generic.h>
> #include <net/vxlan.h>
> #include <net/protocol.h>
> +#include <net/udp_tunnel.h>
> #if IS_ENABLED(CONFIG_IPV6)
> #include <net/ipv6.h>
> #include <net/addrconf.h>
> @@ -277,13 +278,7 @@ static inline struct vxlan_rdst *first_remote_rtnl(struct vxlan_fdb *fdb)
> /* Find VXLAN socket based on network namespace and UDP port */
> static struct vxlan_sock *vxlan_find_sock(struct net *net, __be16 port)
> {
> - struct vxlan_sock *vs;
> -
> - hlist_for_each_entry_rcu(vs, vs_head(net, port), hlist) {
> - if (inet_sk(vs->sock->sk)->inet_sport == port)
> - return vs;
> - }
> - return NULL;
> + return (struct vxlan_sock *)udp_tunnel_find_sock(net, port);
It seems incorrect to assume that the socket returned is vxlan,
presumably this could be some other type of tunnel socket.
> }
>
> static struct vxlan_dev *vxlan_vs_find_vni(struct vxlan_sock *vs, u32 id)
> @@ -636,7 +631,7 @@ static int vxlan_gro_complete(struct sk_buff *skb, int nhoff)
> static void vxlan_notify_add_rx_port(struct vxlan_sock *vs)
> {
> struct net_device *dev;
> - struct sock *sk = vs->sock->sk;
> + struct sock *sk = vs->uts.sock->sk;
> struct net *net = sock_net(sk);
> sa_family_t sa_family = sk->sk_family;
> __be16 port = inet_sk(sk)->inet_sport;
> @@ -661,7 +656,7 @@ static void vxlan_notify_add_rx_port(struct vxlan_sock *vs)
> static void vxlan_notify_del_rx_port(struct vxlan_sock *vs)
> {
> struct net_device *dev;
> - struct sock *sk = vs->sock->sk;
> + struct sock *sk = vs->uts.sock->sk;
> struct net *net = sock_net(sk);
> sa_family_t sa_family = sk->sk_family;
> __be16 port = inet_sk(sk)->inet_sport;
> @@ -1053,7 +1048,7 @@ static void vxlan_sock_hold(struct vxlan_sock *vs)
>
> void vxlan_sock_release(struct vxlan_sock *vs)
> {
> - struct sock *sk = vs->sock->sk;
> + struct sock *sk = vs->uts.sock->sk;
> struct net *net = sock_net(sk);
> struct vxlan_net *vn = net_generic(net, vxlan_net_id);
>
> @@ -1062,7 +1057,6 @@ void vxlan_sock_release(struct vxlan_sock *vs)
>
> spin_lock(&vn->sock_lock);
> hlist_del_rcu(&vs->hlist);
> - rcu_assign_sk_user_data(vs->sock->sk, NULL);
> vxlan_notify_del_rx_port(vs);
> spin_unlock(&vn->sock_lock);
>
> @@ -1078,7 +1072,7 @@ static void vxlan_igmp_join(struct work_struct *work)
> {
> struct vxlan_dev *vxlan = container_of(work, struct vxlan_dev, igmp_join);
> struct vxlan_sock *vs = vxlan->vn_sock;
> - struct sock *sk = vs->sock->sk;
> + struct sock *sk = vs->uts.sock->sk;
> union vxlan_addr *ip = &vxlan->default_dst.remote_ip;
> int ifindex = vxlan->default_dst.remote_ifindex;
>
> @@ -1107,7 +1101,7 @@ static void vxlan_igmp_leave(struct work_struct *work)
> {
> struct vxlan_dev *vxlan = container_of(work, struct vxlan_dev, igmp_leave);
> struct vxlan_sock *vs = vxlan->vn_sock;
> - struct sock *sk = vs->sock->sk;
> + struct sock *sk = vs->uts.sock->sk;
> union vxlan_addr *ip = &vxlan->default_dst.remote_ip;
> int ifindex = vxlan->default_dst.remote_ifindex;
>
> @@ -1338,7 +1332,6 @@ out:
> }
>
> #if IS_ENABLED(CONFIG_IPV6)
> -
> static struct sk_buff *vxlan_na_create(struct sk_buff *request,
> struct neighbour *n, bool isrouter)
> {
> @@ -1572,13 +1565,6 @@ static bool route_shortcircuit(struct net_device *dev, struct sk_buff *skb)
> return false;
> }
>
> -static inline struct sk_buff *vxlan_handle_offloads(struct sk_buff *skb,
> - bool udp_csum)
> -{
> - int type = udp_csum ? SKB_GSO_UDP_TUNNEL_CSUM : SKB_GSO_UDP_TUNNEL;
> - return iptunnel_handle_offloads(skb, udp_csum, type);
> -}
> -
> #if IS_ENABLED(CONFIG_IPV6)
> static int vxlan6_xmit_skb(struct vxlan_sock *vs,
> struct dst_entry *dst, struct sk_buff *skb,
> @@ -1587,13 +1573,13 @@ static int vxlan6_xmit_skb(struct vxlan_sock *vs,
> __be16 src_port, __be16 dst_port, __be32 vni,
> bool xnet)
> {
> - struct ipv6hdr *ip6h;
> struct vxlanhdr *vxh;
> - struct udphdr *uh;
> int min_headroom;
> int err;
>
> - skb = vxlan_handle_offloads(skb, !udp_get_no_check6_tx(vs->sock->sk));
> + skb = udp_tunnel_handle_offloads(skb,
> + !udp_get_no_check6_tx(
> + vs->uts.sock->sk));
> if (IS_ERR(skb))
> return -EINVAL;
>
> @@ -1621,38 +1607,8 @@ static int vxlan6_xmit_skb(struct vxlan_sock *vs,
> vxh->vx_flags = htonl(VXLAN_FLAGS);
> vxh->vx_vni = vni;
>
> - __skb_push(skb, sizeof(*uh));
> - skb_reset_transport_header(skb);
> - uh = udp_hdr(skb);
> -
> - uh->dest = dst_port;
> - uh->source = src_port;
> -
> - uh->len = htons(skb->len);
> -
> - memset(&(IPCB(skb)->opt), 0, sizeof(IPCB(skb)->opt));
> - IPCB(skb)->flags &= ~(IPSKB_XFRM_TUNNEL_SIZE | IPSKB_XFRM_TRANSFORMED |
> - IPSKB_REROUTED);
> - skb_dst_set(skb, dst);
> -
> - udp6_set_csum(udp_get_no_check6_tx(vs->sock->sk), skb,
> - saddr, daddr, skb->len);
> -
> - __skb_push(skb, sizeof(*ip6h));
> - skb_reset_network_header(skb);
> - ip6h = ipv6_hdr(skb);
> - ip6h->version = 6;
> - ip6h->priority = prio;
> - ip6h->flow_lbl[0] = 0;
> - ip6h->flow_lbl[1] = 0;
> - ip6h->flow_lbl[2] = 0;
> - ip6h->payload_len = htons(skb->len);
> - ip6h->nexthdr = IPPROTO_UDP;
> - ip6h->hop_limit = ttl;
> - ip6h->daddr = *daddr;
> - ip6h->saddr = *saddr;
> -
> - ip6tunnel_xmit(skb, dev);
> + udp_tunnel6_xmit_skb(vs->uts.sock, dst, skb, dev, saddr, daddr, prio,
> + ttl, src_port, dst_port);
> return 0;
> }
> #endif
> @@ -1663,11 +1619,11 @@ int vxlan_xmit_skb(struct vxlan_sock *vs,
> __be16 src_port, __be16 dst_port, __be32 vni, bool xnet)
> {
> struct vxlanhdr *vxh;
> - struct udphdr *uh;
> int min_headroom;
> int err;
>
> - skb = vxlan_handle_offloads(skb, !vs->sock->sk->sk_no_check_tx);
> + skb = udp_tunnel_handle_offloads(skb,
> + !vs->uts.sock->sk->sk_no_check_tx);
> if (IS_ERR(skb))
> return -EINVAL;
>
> @@ -1693,20 +1649,8 @@ int vxlan_xmit_skb(struct vxlan_sock *vs,
> vxh->vx_flags = htonl(VXLAN_FLAGS);
> vxh->vx_vni = vni;
>
> - __skb_push(skb, sizeof(*uh));
> - skb_reset_transport_header(skb);
> - uh = udp_hdr(skb);
> -
> - uh->dest = dst_port;
> - uh->source = src_port;
> -
> - uh->len = htons(skb->len);
> -
> - udp_set_csum(vs->sock->sk->sk_no_check_tx, skb,
> - src, dst, skb->len);
> -
> - return iptunnel_xmit(vs->sock->sk, rt, skb, src, dst, IPPROTO_UDP,
> - tos, ttl, df, xnet);
> + return udp_tunnel_xmit_skb(vs->uts.sock, rt, skb, src, dst, tos,
> + ttl, df, src_port, dst_port, xnet);
> }
> EXPORT_SYMBOL_GPL(vxlan_xmit_skb);
>
> @@ -1831,18 +1775,18 @@ static void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev,
> tos = ip_tunnel_ecn_encap(tos, old_iph, skb);
> ttl = ttl ? : ip4_dst_hoplimit(&rt->dst);
>
> - err = vxlan_xmit_skb(vxlan->vn_sock, rt, skb,
> - fl4.saddr, dst->sin.sin_addr.s_addr,
> - tos, ttl, df, src_port, dst_port,
> - htonl(vni << 8),
> - !net_eq(vxlan->net, dev_net(vxlan->dev)));
> + err = udp_tunnel_xmit_skb(vxlan->vn_sock->uts.sock, rt, skb,
> + fl4.saddr, dst->sin.sin_addr.s_addr,
> + tos, ttl, df, src_port, dst_port,
> + !net_eq(vxlan->net,
> + dev_net(vxlan->dev)));
>
> if (err < 0)
> goto rt_tx_error;
> iptunnel_xmit_stats(err, &dev->stats, dev->tstats);
> #if IS_ENABLED(CONFIG_IPV6)
> } else {
> - struct sock *sk = vxlan->vn_sock->sock->sk;
> + struct sock *sk = vxlan->vn_sock->uts.sock->sk;
> struct dst_entry *ndst;
> struct flowi6 fl6;
> u32 flags;
> @@ -2204,8 +2148,8 @@ void vxlan_get_rx_port(struct net_device *dev)
> spin_lock(&vn->sock_lock);
> for (i = 0; i < PORT_HASH_SIZE; ++i) {
> hlist_for_each_entry_rcu(vs, &vn->sock_list[i], hlist) {
> - port = inet_sk(vs->sock->sk)->inet_sport;
> - sa_family = vs->sock->sk->sk_family;
> + port = inet_sk(vs->uts.sock->sk)->inet_sport;
> + sa_family = vs->uts.sock->sk->sk_family;
> dev->netdev_ops->ndo_add_vxlan_port(dev, sa_family,
> port);
> }
> @@ -2335,79 +2279,60 @@ static const struct ethtool_ops vxlan_ethtool_ops = {
> static void vxlan_del_work(struct work_struct *work)
> {
> struct vxlan_sock *vs = container_of(work, struct vxlan_sock, del_work);
> -
> - sk_release_kernel(vs->sock->sk);
> + udp_tunnel_sock_release(&vs->uts);
> kfree_rcu(vs, rcu);
> }
>
> -static struct socket *vxlan_create_sock(struct net *net, bool ipv6,
> - __be16 port, u32 flags)
> +/* Create new listen socket if needed */
> +static struct vxlan_sock *vxlan_socket_create(struct net *net, __be16 port,
> + vxlan_rcv_t rcv, void *data,
> + u32 flags)
> {
> - struct socket *sock;
> - struct udp_port_cfg udp_conf;
> - int err;
> + struct vxlan_net *vn = net_generic(net, vxlan_net_id);
> + struct vxlan_sock *vs;
> + struct udp_tunnel_socket_cfg vxlan_ts_cfg;
> + bool ipv6 = !!(flags & VXLAN_F_IPV6);
> + unsigned int h;
>
> - memset(&udp_conf, 0, sizeof(udp_conf));
> + memset(&vxlan_ts_cfg, 0, sizeof(struct udp_tunnel_socket_cfg));
>
> if (ipv6) {
> - udp_conf.family = AF_INET6;
> - udp_conf.use_udp6_tx_checksums =
> + vxlan_ts_cfg.port.family = AF_INET6;
> + vxlan_ts_cfg.port.use_udp6_tx_checksums =
> !!(flags & VXLAN_F_UDP_ZERO_CSUM6_TX);
> - udp_conf.use_udp6_rx_checksums =
> + vxlan_ts_cfg.port.use_udp6_rx_checksums =
> !!(flags & VXLAN_F_UDP_ZERO_CSUM6_RX);
> } else {
> - udp_conf.family = AF_INET;
> - udp_conf.local_ip.s_addr = INADDR_ANY;
> - udp_conf.use_udp_checksums =
> + vxlan_ts_cfg.port.family = AF_INET;
> + vxlan_ts_cfg.port.local_ip.s_addr = INADDR_ANY;
> + vxlan_ts_cfg.port.use_udp_checksums =
> !!(flags & VXLAN_F_UDP_CSUM);
> }
>
> - udp_conf.local_udp_port = port;
> + vxlan_ts_cfg.port.local_udp_port = port;
> + vxlan_ts_cfg.encap_type = 1;
> + vxlan_ts_cfg.encap_rcv = vxlan_udp_encap_recv;
> + vxlan_ts_cfg.encap_destroy = NULL;
>
> - /* Open UDP socket */
> - err = udp_sock_create(net, &udp_conf, &sock);
> - if (err < 0)
> - return ERR_PTR(err);
> -
> - /* Disable multicast loopback */
> - inet_sk(sock->sk)->mc_loop = 0;
> -
> - return sock;
> -}
> -
> -/* Create new listen socket if needed */
> -static struct vxlan_sock *vxlan_socket_create(struct net *net, __be16 port,
> - vxlan_rcv_t *rcv, void *data,
> - u32 flags)
> -{
> - struct vxlan_net *vn = net_generic(net, vxlan_net_id);
> - struct vxlan_sock *vs;
> - struct socket *sock;
> - struct sock *sk;
> - unsigned int h;
> - bool ipv6 = !!(flags & VXLAN_F_IPV6);
> -
> - vs = kzalloc(sizeof(*vs), GFP_KERNEL);
> + vs = (struct vxlan_sock *)create_udp_tunnel_sock(net, sizeof(*vs),
> + NULL,
> + &vxlan_ts_cfg);
> if (!vs)
> return ERR_PTR(-ENOMEM);
>
> for (h = 0; h < VNI_HASH_SIZE; ++h)
> INIT_HLIST_HEAD(&vs->vni_list[h]);
>
> - INIT_WORK(&vs->del_work, vxlan_del_work);
> + spin_lock(&vn->sock_lock);
> + list_add(&vs->next, &vn->vxlan_list);
> + spin_unlock(&vn->sock_lock);
>
> - sock = vxlan_create_sock(net, ipv6, port, flags);
> - if (IS_ERR(sock)) {
> - kfree(vs);
> - return ERR_CAST(sock);
> - }
> + INIT_WORK(&vs->del_work, vxlan_del_work);
>
> - vs->sock = sock;
> - sk = sock->sk;
> atomic_set(&vs->refcnt, 1);
> +
> vs->rcv = rcv;
> - vs->data = data;
> - rcu_assign_sk_user_data(vs->sock->sk, vs);
> + vs->rcv_data = data;
>
> /* Initialize the vxlan udp offloads structure */
> vs->udp_offloads.port = port;
> @@ -2419,24 +2344,13 @@ static struct vxlan_sock *vxlan_socket_create(struct net *net, __be16 port,
> vxlan_notify_add_rx_port(vs);
> spin_unlock(&vn->sock_lock);
>
> - /* Mark socket as an encapsulation socket. */
> - udp_sk(sk)->encap_type = 1;
> - udp_sk(sk)->encap_rcv = vxlan_udp_encap_recv;
> -#if IS_ENABLED(CONFIG_IPV6)
> - if (ipv6)
> - ipv6_stub->udpv6_encap_enable();
> - else
> -#endif
> - udp_encap_enable();
> -
> return vs;
> }
>
> struct vxlan_sock *vxlan_sock_add(struct net *net, __be16 port,
> - vxlan_rcv_t *rcv, void *data,
> + vxlan_rcv_t rcv, void *data,
> bool no_share, u32 flags)
> {
> - struct vxlan_net *vn = net_generic(net, vxlan_net_id);
> struct vxlan_sock *vs;
>
> vs = vxlan_socket_create(net, port, rcv, data, flags);
> @@ -2446,7 +2360,6 @@ struct vxlan_sock *vxlan_sock_add(struct net *net, __be16 port,
> if (no_share) /* Return error if sharing is not allowed. */
> return vs;
>
> - spin_lock(&vn->sock_lock);
> vs = vxlan_find_sock(net, port);
> if (vs) {
> if (vs->rcv == rcv)
> @@ -2454,7 +2367,6 @@ struct vxlan_sock *vxlan_sock_add(struct net *net, __be16 port,
> else
> vs = ERR_PTR(-EBUSY);
> }
> - spin_unlock(&vn->sock_lock);
>
> if (!vs)
> vs = ERR_PTR(-EINVAL);
> @@ -2634,7 +2546,6 @@ static int vxlan_newlink(struct net *net, struct net_device *dev,
> }
>
> list_add(&vxlan->next, &vn->vxlan_list);
> -
> return 0;
> }
>
> diff --git a/include/net/vxlan.h b/include/net/vxlan.h
> index d5f59f3..10bfc13 100644
> --- a/include/net/vxlan.h
> +++ b/include/net/vxlan.h
> @@ -4,23 +4,27 @@
> #include <linux/skbuff.h>
> #include <linux/netdevice.h>
> #include <linux/udp.h>
> +#include <net/udp_tunnel.h>
>
> #define VNI_HASH_BITS 10
> #define VNI_HASH_SIZE (1<<VNI_HASH_BITS)
>
> struct vxlan_sock;
> -typedef void (vxlan_rcv_t)(struct vxlan_sock *vh, struct sk_buff *skb, __be32 key);
>
> -/* per UDP socket information */
> +typedef void (*vxlan_rcv_t)(struct vxlan_sock *vs, struct sk_buff *skb,
> + __be32 key);
> +
> +/* per vxlan socket information */
> struct vxlan_sock {
> + struct udp_tunnel_sock uts; /* Must be the first member */
> struct hlist_node hlist;
> - vxlan_rcv_t *rcv;
> - void *data;
> + struct list_head next;
> struct work_struct del_work;
> - struct socket *sock;
> struct rcu_head rcu;
> struct hlist_head vni_list[VNI_HASH_SIZE];
> atomic_t refcnt;
> + vxlan_rcv_t rcv;
> + void *rcv_data;
> struct udp_offload udp_offloads;
> };
>
> @@ -35,7 +39,7 @@ struct vxlan_sock {
> #define VXLAN_F_UDP_ZERO_CSUM6_RX 0x100
>
> struct vxlan_sock *vxlan_sock_add(struct net *net, __be16 port,
> - vxlan_rcv_t *rcv, void *data,
> + vxlan_rcv_t rcv, void *data,
> bool no_share, u32 flags);
>
> void vxlan_sock_release(struct vxlan_sock *vs);
> diff --git a/net/openvswitch/vport-vxlan.c b/net/openvswitch/vport-vxlan.c
> index d8b7e24..7599efd 100644
> --- a/net/openvswitch/vport-vxlan.c
> +++ b/net/openvswitch/vport-vxlan.c
> @@ -59,7 +59,7 @@ static inline struct vxlan_port *vxlan_vport(const struct vport *vport)
> static void vxlan_rcv(struct vxlan_sock *vs, struct sk_buff *skb, __be32 vx_vni)
> {
> struct ovs_key_ipv4_tunnel tun_key;
> - struct vport *vport = vs->data;
> + struct vport *vport = vs->rcv_data;
> struct iphdr *iph;
> __be64 key;
>
> @@ -74,7 +74,7 @@ static void vxlan_rcv(struct vxlan_sock *vs, struct sk_buff *skb, __be32 vx_vni)
> static int vxlan_get_options(const struct vport *vport, struct sk_buff *skb)
> {
> struct vxlan_port *vxlan_port = vxlan_vport(vport);
> - __be16 dst_port = inet_sk(vxlan_port->vs->sock->sk)->inet_sport;
> + __be16 dst_port = inet_sk(vxlan_port->vs->uts.sock->sk)->inet_sport;
>
> if (nla_put_u16(skb, OVS_TUNNEL_ATTR_DST_PORT, ntohs(dst_port)))
> return -EMSGSIZE;
> @@ -139,7 +139,7 @@ static int vxlan_tnl_send(struct vport *vport, struct sk_buff *skb)
> {
> struct net *net = ovs_dp_get_net(vport->dp);
> struct vxlan_port *vxlan_port = vxlan_vport(vport);
> - __be16 dst_port = inet_sk(vxlan_port->vs->sock->sk)->inet_sport;
> + __be16 dst_port = inet_sk(vxlan_port->vs->uts.sock->sk)->inet_sport;
> struct rtable *rt;
> struct flowi4 fl;
> __be16 src_port;
> --
> 1.7.9.5
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCHv2 00/14] arm64: eBPF JIT compiler
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, Jiang Liu, AKASHI Takahiro, David S. Miller,
Daniel Borkmann, Alexei Starovoitov, Chema Gonzalez, linux-kernel,
linux-arm-kernel, netdev
Hi Will, Catalin,
This is a respin of series implementing eBPF JIT compiler for arm64,
on top of 3.17-rc2. The v1 series [1] missed the previous merge window.
Patches [1-13/14] implement code generation functions. Unchanged from v1.
Patch [14/14] implements the actual eBPF JIT compiler. Updated from v1:
straightforward fixups due to changes from net.
Please see [14/14] for detailed change log.
This series is applies cleanly against 3.17-rc2 and is tested working
with lib/test_bpf on ARMv8 Foundation Model.
Will had previously reported that v1 series works on Juno platform.
Since v2 only involves straightforward renaming in [14/14], I don't
anticipate any regressions.
Thanks,
z
[1] https://lkml.org/lkml/2014/7/18/683
--------
The following changes since commit 52addcf9d6669fa439387610bc65c92fa0980cef:
Linux 3.17-rc2 (2014-08-25 15:36:20 -0700)
are available in the git repository at:
https://github.com/zlim/linux.git tags/arm64/bpf-v2
for you to fetch changes up to 2f4a4b8df4ba1cbd24957fb1a8371d30b1976174:
arm64: eBPF JIT compiler (2014-08-26 19:04:43 -0700)
----------------------------------------------------------------
Documentation/networking/filter.txt | 6 +-
arch/arm64/Kconfig | 1 +
arch/arm64/Makefile | 1 +
arch/arm64/include/asm/insn.h | 249 +++++++++++++
arch/arm64/kernel/insn.c | 646 +++++++++++++++++++++++++++++++++-
arch/arm64/net/Makefile | 4 +
arch/arm64/net/bpf_jit.h | 169 +++++++++
arch/arm64/net/bpf_jit_comp.c | 677 ++++++++++++++++++++++++++++++++++++
8 files changed, 1743 insertions(+), 10 deletions(-)
create mode 100644 arch/arm64/net/Makefile
create mode 100644 arch/arm64/net/bpf_jit.h
create mode 100644 arch/arm64/net/bpf_jit_comp.c
Zi Shen Lim (14):
arm64: introduce aarch64_insn_gen_comp_branch_imm()
arm64: introduce aarch64_insn_gen_branch_reg()
arm64: introduce aarch64_insn_gen_cond_branch_imm()
arm64: introduce aarch64_insn_gen_load_store_reg()
arm64: introduce aarch64_insn_gen_load_store_pair()
arm64: introduce aarch64_insn_gen_add_sub_imm()
arm64: introduce aarch64_insn_gen_bitfield()
arm64: introduce aarch64_insn_gen_movewide()
arm64: introduce aarch64_insn_gen_add_sub_shifted_reg()
arm64: introduce aarch64_insn_gen_data1()
arm64: introduce aarch64_insn_gen_data2()
arm64: introduce aarch64_insn_gen_data3()
arm64: introduce aarch64_insn_gen_logical_shifted_reg()
arm64: eBPF JIT compiler
Documentation/networking/filter.txt | 6 +-
arch/arm64/Kconfig | 1 +
arch/arm64/Makefile | 1 +
arch/arm64/include/asm/insn.h | 249 +++++++++++++
arch/arm64/kernel/insn.c | 646 +++++++++++++++++++++++++++++++++-
arch/arm64/net/Makefile | 4 +
arch/arm64/net/bpf_jit.h | 169 +++++++++
arch/arm64/net/bpf_jit_comp.c | 677 ++++++++++++++++++++++++++++++++++++
8 files changed, 1743 insertions(+), 10 deletions(-)
create mode 100644 arch/arm64/net/Makefile
create mode 100644 arch/arm64/net/bpf_jit.h
create mode 100644 arch/arm64/net/bpf_jit_comp.c
--
1.9.1
^ permalink raw reply
* [PATCHv2 01/14] arm64: introduce aarch64_insn_gen_comp_branch_imm()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, Jiang Liu, AKASHI Takahiro, David S. Miller,
Daniel Borkmann, Alexei Starovoitov, linux-kernel,
linux-arm-kernel, netdev
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate compare & branch (immediate)
instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 57 ++++++++++++++++++++++++++++
arch/arm64/kernel/insn.c | 88 ++++++++++++++++++++++++++++++++++++++++---
2 files changed, 140 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index dc1f73b..a98c495 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -2,6 +2,8 @@
* Copyright (C) 2013 Huawei Ltd.
* Author: Jiang Liu <liuj97@gmail.com>
*
+ * Copyright (C) 2014 Zi Shen Lim <zlim.lnx@gmail.com>
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
@@ -67,9 +69,58 @@ enum aarch64_insn_imm_type {
AARCH64_INSN_IMM_MAX
};
+enum aarch64_insn_register_type {
+ AARCH64_INSN_REGTYPE_RT,
+};
+
+enum aarch64_insn_register {
+ AARCH64_INSN_REG_0 = 0,
+ AARCH64_INSN_REG_1 = 1,
+ AARCH64_INSN_REG_2 = 2,
+ AARCH64_INSN_REG_3 = 3,
+ AARCH64_INSN_REG_4 = 4,
+ AARCH64_INSN_REG_5 = 5,
+ AARCH64_INSN_REG_6 = 6,
+ AARCH64_INSN_REG_7 = 7,
+ AARCH64_INSN_REG_8 = 8,
+ AARCH64_INSN_REG_9 = 9,
+ AARCH64_INSN_REG_10 = 10,
+ AARCH64_INSN_REG_11 = 11,
+ AARCH64_INSN_REG_12 = 12,
+ AARCH64_INSN_REG_13 = 13,
+ AARCH64_INSN_REG_14 = 14,
+ AARCH64_INSN_REG_15 = 15,
+ AARCH64_INSN_REG_16 = 16,
+ AARCH64_INSN_REG_17 = 17,
+ AARCH64_INSN_REG_18 = 18,
+ AARCH64_INSN_REG_19 = 19,
+ AARCH64_INSN_REG_20 = 20,
+ AARCH64_INSN_REG_21 = 21,
+ AARCH64_INSN_REG_22 = 22,
+ AARCH64_INSN_REG_23 = 23,
+ AARCH64_INSN_REG_24 = 24,
+ AARCH64_INSN_REG_25 = 25,
+ AARCH64_INSN_REG_26 = 26,
+ AARCH64_INSN_REG_27 = 27,
+ AARCH64_INSN_REG_28 = 28,
+ AARCH64_INSN_REG_29 = 29,
+ AARCH64_INSN_REG_FP = 29, /* Frame pointer */
+ AARCH64_INSN_REG_30 = 30,
+ AARCH64_INSN_REG_LR = 30, /* Link register */
+ AARCH64_INSN_REG_ZR = 31, /* Zero: as source register */
+ AARCH64_INSN_REG_SP = 31 /* Stack pointer: as load/store base reg */
+};
+
+enum aarch64_insn_variant {
+ AARCH64_INSN_VARIANT_32BIT,
+ AARCH64_INSN_VARIANT_64BIT
+};
+
enum aarch64_insn_branch_type {
AARCH64_INSN_BRANCH_NOLINK,
AARCH64_INSN_BRANCH_LINK,
+ AARCH64_INSN_BRANCH_COMP_ZERO,
+ AARCH64_INSN_BRANCH_COMP_NONZERO,
};
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
@@ -80,6 +131,8 @@ static __always_inline u32 aarch64_insn_get_##abbr##_value(void) \
__AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
+__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
+__AARCH64_INSN_FUNCS(cbnz, 0xFE000000, 0x35000000)
__AARCH64_INSN_FUNCS(svc, 0xFFE0001F, 0xD4000001)
__AARCH64_INSN_FUNCS(hvc, 0xFFE0001F, 0xD4000002)
__AARCH64_INSN_FUNCS(smc, 0xFFE0001F, 0xD4000003)
@@ -97,6 +150,10 @@ u32 aarch64_insn_encode_immediate(enum aarch64_insn_imm_type type,
u32 insn, u64 imm);
u32 aarch64_insn_gen_branch_imm(unsigned long pc, unsigned long addr,
enum aarch64_insn_branch_type type);
+u32 aarch64_insn_gen_comp_branch_imm(unsigned long pc, unsigned long addr,
+ enum aarch64_insn_register reg,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_branch_type type);
u32 aarch64_insn_gen_hint(enum aarch64_insn_hint_op op);
u32 aarch64_insn_gen_nop(void);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 92f3683..d9f7827 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -2,6 +2,8 @@
* Copyright (C) 2013 Huawei Ltd.
* Author: Jiang Liu <liuj97@gmail.com>
*
+ * Copyright (C) 2014 Zi Shen Lim <zlim.lnx@gmail.com>
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
@@ -23,6 +25,8 @@
#include <asm/cacheflush.h>
#include <asm/insn.h>
+#define AARCH64_INSN_SF_BIT BIT(31)
+
static int aarch64_insn_encoding_class[] = {
AARCH64_INSN_CLS_UNKNOWN,
AARCH64_INSN_CLS_UNKNOWN,
@@ -264,10 +268,36 @@ u32 __kprobes aarch64_insn_encode_immediate(enum aarch64_insn_imm_type type,
return insn;
}
-u32 __kprobes aarch64_insn_gen_branch_imm(unsigned long pc, unsigned long addr,
- enum aarch64_insn_branch_type type)
+static u32 aarch64_insn_encode_register(enum aarch64_insn_register_type type,
+ u32 insn,
+ enum aarch64_insn_register reg)
+{
+ int shift;
+
+ if (reg < AARCH64_INSN_REG_0 || reg > AARCH64_INSN_REG_SP) {
+ pr_err("%s: unknown register encoding %d\n", __func__, reg);
+ return 0;
+ }
+
+ switch (type) {
+ case AARCH64_INSN_REGTYPE_RT:
+ shift = 0;
+ break;
+ default:
+ pr_err("%s: unknown register type encoding %d\n", __func__,
+ type);
+ return 0;
+ }
+
+ insn &= ~(GENMASK(4, 0) << shift);
+ insn |= reg << shift;
+
+ return insn;
+}
+
+static inline long branch_imm_common(unsigned long pc, unsigned long addr,
+ long range)
{
- u32 insn;
long offset;
/*
@@ -276,13 +306,24 @@ u32 __kprobes aarch64_insn_gen_branch_imm(unsigned long pc, unsigned long addr,
*/
BUG_ON((pc & 0x3) || (addr & 0x3));
+ offset = ((long)addr - (long)pc);
+ BUG_ON(offset < -range || offset >= range);
+
+ return offset;
+}
+
+u32 __kprobes aarch64_insn_gen_branch_imm(unsigned long pc, unsigned long addr,
+ enum aarch64_insn_branch_type type)
+{
+ u32 insn;
+ long offset;
+
/*
* B/BL support [-128M, 128M) offset
* ARM64 virtual address arrangement guarantees all kernel and module
* texts are within +/-128M.
*/
- offset = ((long)addr - (long)pc);
- BUG_ON(offset < -SZ_128M || offset >= SZ_128M);
+ offset = branch_imm_common(pc, addr, SZ_128M);
if (type == AARCH64_INSN_BRANCH_LINK)
insn = aarch64_insn_get_bl_value();
@@ -293,6 +334,43 @@ u32 __kprobes aarch64_insn_gen_branch_imm(unsigned long pc, unsigned long addr,
offset >> 2);
}
+u32 aarch64_insn_gen_comp_branch_imm(unsigned long pc, unsigned long addr,
+ enum aarch64_insn_register reg,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_branch_type type)
+{
+ u32 insn;
+ long offset;
+
+ offset = branch_imm_common(pc, addr, SZ_1M);
+
+ switch (type) {
+ case AARCH64_INSN_BRANCH_COMP_ZERO:
+ insn = aarch64_insn_get_cbz_value();
+ break;
+ case AARCH64_INSN_BRANCH_COMP_NONZERO:
+ insn = aarch64_insn_get_cbnz_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ insn |= AARCH64_INSN_SF_BIT;
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT, insn, reg);
+
+ return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_19, insn,
+ offset >> 2);
+}
+
u32 __kprobes aarch64_insn_gen_hint(enum aarch64_insn_hint_op op)
{
return aarch64_insn_get_hint_value() | op;
--
1.9.1
^ permalink raw reply related
* [PATCHv2 03/14] arm64: introduce aarch64_insn_gen_cond_branch_imm()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, linux-kernel, AKASHI Takahiro, Daniel Borkmann,
netdev, Jiang Liu, David S. Miller, linux-arm-kernel,
Alexei Starovoitov
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate conditional branch (immediate)
instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 21 +++++++++++++++++++++
arch/arm64/kernel/insn.c | 17 +++++++++++++++++
2 files changed, 38 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 5080962..86a8a9c 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -117,6 +117,24 @@ enum aarch64_insn_variant {
AARCH64_INSN_VARIANT_64BIT
};
+enum aarch64_insn_condition {
+ AARCH64_INSN_COND_EQ = 0x0, /* == */
+ AARCH64_INSN_COND_NE = 0x1, /* != */
+ AARCH64_INSN_COND_CS = 0x2, /* unsigned >= */
+ AARCH64_INSN_COND_CC = 0x3, /* unsigned < */
+ AARCH64_INSN_COND_MI = 0x4, /* < 0 */
+ AARCH64_INSN_COND_PL = 0x5, /* >= 0 */
+ AARCH64_INSN_COND_VS = 0x6, /* overflow */
+ AARCH64_INSN_COND_VC = 0x7, /* no overflow */
+ AARCH64_INSN_COND_HI = 0x8, /* unsigned > */
+ AARCH64_INSN_COND_LS = 0x9, /* unsigned <= */
+ AARCH64_INSN_COND_GE = 0xa, /* signed >= */
+ AARCH64_INSN_COND_LT = 0xb, /* signed < */
+ AARCH64_INSN_COND_GT = 0xc, /* signed > */
+ AARCH64_INSN_COND_LE = 0xd, /* signed <= */
+ AARCH64_INSN_COND_AL = 0xe, /* always */
+};
+
enum aarch64_insn_branch_type {
AARCH64_INSN_BRANCH_NOLINK,
AARCH64_INSN_BRANCH_LINK,
@@ -135,6 +153,7 @@ __AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
__AARCH64_INSN_FUNCS(cbnz, 0xFE000000, 0x35000000)
+__AARCH64_INSN_FUNCS(bcond, 0xFF000010, 0x54000000)
__AARCH64_INSN_FUNCS(svc, 0xFFE0001F, 0xD4000001)
__AARCH64_INSN_FUNCS(hvc, 0xFFE0001F, 0xD4000002)
__AARCH64_INSN_FUNCS(smc, 0xFFE0001F, 0xD4000003)
@@ -159,6 +178,8 @@ u32 aarch64_insn_gen_comp_branch_imm(unsigned long pc, unsigned long addr,
enum aarch64_insn_register reg,
enum aarch64_insn_variant variant,
enum aarch64_insn_branch_type type);
+u32 aarch64_insn_gen_cond_branch_imm(unsigned long pc, unsigned long addr,
+ enum aarch64_insn_condition cond);
u32 aarch64_insn_gen_hint(enum aarch64_insn_hint_op op);
u32 aarch64_insn_gen_nop(void);
u32 aarch64_insn_gen_branch_reg(enum aarch64_insn_register reg,
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 6797936..b65edc0 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -380,6 +380,23 @@ u32 aarch64_insn_gen_comp_branch_imm(unsigned long pc, unsigned long addr,
offset >> 2);
}
+u32 aarch64_insn_gen_cond_branch_imm(unsigned long pc, unsigned long addr,
+ enum aarch64_insn_condition cond)
+{
+ u32 insn;
+ long offset;
+
+ offset = branch_imm_common(pc, addr, SZ_1M);
+
+ insn = aarch64_insn_get_bcond_value();
+
+ BUG_ON(cond < AARCH64_INSN_COND_EQ || cond > AARCH64_INSN_COND_AL);
+ insn |= cond;
+
+ return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_19, insn,
+ offset >> 2);
+}
+
u32 __kprobes aarch64_insn_gen_hint(enum aarch64_insn_hint_op op)
{
return aarch64_insn_get_hint_value() | op;
--
1.9.1
^ permalink raw reply related
* [PATCHv2 04/14] arm64: introduce aarch64_insn_gen_load_store_reg()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, linux-kernel, AKASHI Takahiro, Daniel Borkmann,
netdev, Jiang Liu, David S. Miller, linux-arm-kernel,
Alexei Starovoitov
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate load/store (register offset)
instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 20 ++++++++++++++
arch/arm64/kernel/insn.c | 62 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 82 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 86a8a9c..5bc1cc3 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -72,6 +72,7 @@ enum aarch64_insn_imm_type {
enum aarch64_insn_register_type {
AARCH64_INSN_REGTYPE_RT,
AARCH64_INSN_REGTYPE_RN,
+ AARCH64_INSN_REGTYPE_RM,
};
enum aarch64_insn_register {
@@ -143,12 +144,26 @@ enum aarch64_insn_branch_type {
AARCH64_INSN_BRANCH_COMP_NONZERO,
};
+enum aarch64_insn_size_type {
+ AARCH64_INSN_SIZE_8,
+ AARCH64_INSN_SIZE_16,
+ AARCH64_INSN_SIZE_32,
+ AARCH64_INSN_SIZE_64,
+};
+
+enum aarch64_insn_ldst_type {
+ AARCH64_INSN_LDST_LOAD_REG_OFFSET,
+ AARCH64_INSN_LDST_STORE_REG_OFFSET,
+};
+
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
static __always_inline bool aarch64_insn_is_##abbr(u32 code) \
{ return (code & (mask)) == (val); } \
static __always_inline u32 aarch64_insn_get_##abbr##_value(void) \
{ return (val); }
+__AARCH64_INSN_FUNCS(str_reg, 0x3FE0EC00, 0x38206800)
+__AARCH64_INSN_FUNCS(ldr_reg, 0x3FE0EC00, 0x38606800)
__AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
@@ -184,6 +199,11 @@ u32 aarch64_insn_gen_hint(enum aarch64_insn_hint_op op);
u32 aarch64_insn_gen_nop(void);
u32 aarch64_insn_gen_branch_reg(enum aarch64_insn_register reg,
enum aarch64_insn_branch_type type);
+u32 aarch64_insn_gen_load_store_reg(enum aarch64_insn_register reg,
+ enum aarch64_insn_register base,
+ enum aarch64_insn_register offset,
+ enum aarch64_insn_size_type size,
+ enum aarch64_insn_ldst_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index b65edc0..b882c85 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -286,6 +286,9 @@ static u32 aarch64_insn_encode_register(enum aarch64_insn_register_type type,
case AARCH64_INSN_REGTYPE_RN:
shift = 5;
break;
+ case AARCH64_INSN_REGTYPE_RM:
+ shift = 16;
+ break;
default:
pr_err("%s: unknown register type encoding %d\n", __func__,
type);
@@ -298,6 +301,35 @@ static u32 aarch64_insn_encode_register(enum aarch64_insn_register_type type,
return insn;
}
+static u32 aarch64_insn_encode_ldst_size(enum aarch64_insn_size_type type,
+ u32 insn)
+{
+ u32 size;
+
+ switch (type) {
+ case AARCH64_INSN_SIZE_8:
+ size = 0;
+ break;
+ case AARCH64_INSN_SIZE_16:
+ size = 1;
+ break;
+ case AARCH64_INSN_SIZE_32:
+ size = 2;
+ break;
+ case AARCH64_INSN_SIZE_64:
+ size = 3;
+ break;
+ default:
+ pr_err("%s: unknown size encoding %d\n", __func__, type);
+ return 0;
+ }
+
+ insn &= ~GENMASK(31, 30);
+ insn |= size << 30;
+
+ return insn;
+}
+
static inline long branch_imm_common(unsigned long pc, unsigned long addr,
long range)
{
@@ -428,3 +460,33 @@ u32 aarch64_insn_gen_branch_reg(enum aarch64_insn_register reg,
return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, reg);
}
+
+u32 aarch64_insn_gen_load_store_reg(enum aarch64_insn_register reg,
+ enum aarch64_insn_register base,
+ enum aarch64_insn_register offset,
+ enum aarch64_insn_size_type size,
+ enum aarch64_insn_ldst_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_LDST_LOAD_REG_OFFSET:
+ insn = aarch64_insn_get_ldr_reg_value();
+ break;
+ case AARCH64_INSN_LDST_STORE_REG_OFFSET:
+ insn = aarch64_insn_get_str_reg_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ insn = aarch64_insn_encode_ldst_size(size, insn);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT, insn, reg);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn,
+ base);
+
+ return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn,
+ offset);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 05/14] arm64: introduce aarch64_insn_gen_load_store_pair()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, linux-kernel, AKASHI Takahiro, Daniel Borkmann,
netdev, Jiang Liu, David S. Miller, linux-arm-kernel,
Alexei Starovoitov
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate load/store pair instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 16 +++++++++++
arch/arm64/kernel/insn.c | 65 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 81 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 5bc1cc3..eef8f1e 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -66,12 +66,14 @@ enum aarch64_insn_imm_type {
AARCH64_INSN_IMM_14,
AARCH64_INSN_IMM_12,
AARCH64_INSN_IMM_9,
+ AARCH64_INSN_IMM_7,
AARCH64_INSN_IMM_MAX
};
enum aarch64_insn_register_type {
AARCH64_INSN_REGTYPE_RT,
AARCH64_INSN_REGTYPE_RN,
+ AARCH64_INSN_REGTYPE_RT2,
AARCH64_INSN_REGTYPE_RM,
};
@@ -154,6 +156,10 @@ enum aarch64_insn_size_type {
enum aarch64_insn_ldst_type {
AARCH64_INSN_LDST_LOAD_REG_OFFSET,
AARCH64_INSN_LDST_STORE_REG_OFFSET,
+ AARCH64_INSN_LDST_LOAD_PAIR_PRE_INDEX,
+ AARCH64_INSN_LDST_STORE_PAIR_PRE_INDEX,
+ AARCH64_INSN_LDST_LOAD_PAIR_POST_INDEX,
+ AARCH64_INSN_LDST_STORE_PAIR_POST_INDEX,
};
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
@@ -164,6 +170,10 @@ static __always_inline u32 aarch64_insn_get_##abbr##_value(void) \
__AARCH64_INSN_FUNCS(str_reg, 0x3FE0EC00, 0x38206800)
__AARCH64_INSN_FUNCS(ldr_reg, 0x3FE0EC00, 0x38606800)
+__AARCH64_INSN_FUNCS(stp_post, 0x7FC00000, 0x28800000)
+__AARCH64_INSN_FUNCS(ldp_post, 0x7FC00000, 0x28C00000)
+__AARCH64_INSN_FUNCS(stp_pre, 0x7FC00000, 0x29800000)
+__AARCH64_INSN_FUNCS(ldp_pre, 0x7FC00000, 0x29C00000)
__AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
@@ -204,6 +214,12 @@ u32 aarch64_insn_gen_load_store_reg(enum aarch64_insn_register reg,
enum aarch64_insn_register offset,
enum aarch64_insn_size_type size,
enum aarch64_insn_ldst_type type);
+u32 aarch64_insn_gen_load_store_pair(enum aarch64_insn_register reg1,
+ enum aarch64_insn_register reg2,
+ enum aarch64_insn_register base,
+ int offset,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_ldst_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index b882c85..7880c06 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -255,6 +255,10 @@ u32 __kprobes aarch64_insn_encode_immediate(enum aarch64_insn_imm_type type,
mask = BIT(9) - 1;
shift = 12;
break;
+ case AARCH64_INSN_IMM_7:
+ mask = BIT(7) - 1;
+ shift = 15;
+ break;
default:
pr_err("aarch64_insn_encode_immediate: unknown immediate encoding %d\n",
type);
@@ -286,6 +290,9 @@ static u32 aarch64_insn_encode_register(enum aarch64_insn_register_type type,
case AARCH64_INSN_REGTYPE_RN:
shift = 5;
break;
+ case AARCH64_INSN_REGTYPE_RT2:
+ shift = 10;
+ break;
case AARCH64_INSN_REGTYPE_RM:
shift = 16;
break;
@@ -490,3 +497,61 @@ u32 aarch64_insn_gen_load_store_reg(enum aarch64_insn_register reg,
return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn,
offset);
}
+
+u32 aarch64_insn_gen_load_store_pair(enum aarch64_insn_register reg1,
+ enum aarch64_insn_register reg2,
+ enum aarch64_insn_register base,
+ int offset,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_ldst_type type)
+{
+ u32 insn;
+ int shift;
+
+ switch (type) {
+ case AARCH64_INSN_LDST_LOAD_PAIR_PRE_INDEX:
+ insn = aarch64_insn_get_ldp_pre_value();
+ break;
+ case AARCH64_INSN_LDST_STORE_PAIR_PRE_INDEX:
+ insn = aarch64_insn_get_stp_pre_value();
+ break;
+ case AARCH64_INSN_LDST_LOAD_PAIR_POST_INDEX:
+ insn = aarch64_insn_get_ldp_post_value();
+ break;
+ case AARCH64_INSN_LDST_STORE_PAIR_POST_INDEX:
+ insn = aarch64_insn_get_stp_post_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ /* offset must be multiples of 4 in the range [-256, 252] */
+ BUG_ON(offset & 0x3);
+ BUG_ON(offset < -256 || offset > 252);
+ shift = 2;
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ /* offset must be multiples of 8 in the range [-512, 504] */
+ BUG_ON(offset & 0x7);
+ BUG_ON(offset < -512 || offset > 504);
+ shift = 3;
+ insn |= AARCH64_INSN_SF_BIT;
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT, insn,
+ reg1);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT2, insn,
+ reg2);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn,
+ base);
+
+ return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_7, insn,
+ offset >> shift);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 06/14] arm64: introduce aarch64_insn_gen_add_sub_imm()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, linux-kernel, AKASHI Takahiro, Daniel Borkmann,
netdev, Jiang Liu, David S. Miller, linux-arm-kernel,
Alexei Starovoitov
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate add/subtract (immediate) instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 16 ++++++++++++++++
arch/arm64/kernel/insn.c | 44 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 60 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index eef8f1e..29386aa 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -75,6 +75,7 @@ enum aarch64_insn_register_type {
AARCH64_INSN_REGTYPE_RN,
AARCH64_INSN_REGTYPE_RT2,
AARCH64_INSN_REGTYPE_RM,
+ AARCH64_INSN_REGTYPE_RD,
};
enum aarch64_insn_register {
@@ -162,6 +163,13 @@ enum aarch64_insn_ldst_type {
AARCH64_INSN_LDST_STORE_PAIR_POST_INDEX,
};
+enum aarch64_insn_adsb_type {
+ AARCH64_INSN_ADSB_ADD,
+ AARCH64_INSN_ADSB_SUB,
+ AARCH64_INSN_ADSB_ADD_SETFLAGS,
+ AARCH64_INSN_ADSB_SUB_SETFLAGS
+};
+
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
static __always_inline bool aarch64_insn_is_##abbr(u32 code) \
{ return (code & (mask)) == (val); } \
@@ -174,6 +182,10 @@ __AARCH64_INSN_FUNCS(stp_post, 0x7FC00000, 0x28800000)
__AARCH64_INSN_FUNCS(ldp_post, 0x7FC00000, 0x28C00000)
__AARCH64_INSN_FUNCS(stp_pre, 0x7FC00000, 0x29800000)
__AARCH64_INSN_FUNCS(ldp_pre, 0x7FC00000, 0x29C00000)
+__AARCH64_INSN_FUNCS(add_imm, 0x7F000000, 0x11000000)
+__AARCH64_INSN_FUNCS(adds_imm, 0x7F000000, 0x31000000)
+__AARCH64_INSN_FUNCS(sub_imm, 0x7F000000, 0x51000000)
+__AARCH64_INSN_FUNCS(subs_imm, 0x7F000000, 0x71000000)
__AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
@@ -220,6 +232,10 @@ u32 aarch64_insn_gen_load_store_pair(enum aarch64_insn_register reg1,
int offset,
enum aarch64_insn_variant variant,
enum aarch64_insn_ldst_type type);
+u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ int imm, enum aarch64_insn_variant variant,
+ enum aarch64_insn_adsb_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 7880c06..ec3a902 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -285,6 +285,7 @@ static u32 aarch64_insn_encode_register(enum aarch64_insn_register_type type,
switch (type) {
case AARCH64_INSN_REGTYPE_RT:
+ case AARCH64_INSN_REGTYPE_RD:
shift = 0;
break;
case AARCH64_INSN_REGTYPE_RN:
@@ -555,3 +556,46 @@ u32 aarch64_insn_gen_load_store_pair(enum aarch64_insn_register reg1,
return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_7, insn,
offset >> shift);
}
+
+u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ int imm, enum aarch64_insn_variant variant,
+ enum aarch64_insn_adsb_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_ADSB_ADD:
+ insn = aarch64_insn_get_add_imm_value();
+ break;
+ case AARCH64_INSN_ADSB_SUB:
+ insn = aarch64_insn_get_sub_imm_value();
+ break;
+ case AARCH64_INSN_ADSB_ADD_SETFLAGS:
+ insn = aarch64_insn_get_adds_imm_value();
+ break;
+ case AARCH64_INSN_ADSB_SUB_SETFLAGS:
+ insn = aarch64_insn_get_subs_imm_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ insn |= AARCH64_INSN_SF_BIT;
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ BUG_ON(imm & ~(SZ_4K - 1));
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
+
+ return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_12, insn, imm);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 07/14] arm64: introduce aarch64_insn_gen_bitfield()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, Jiang Liu, AKASHI Takahiro, David S. Miller,
Daniel Borkmann, Alexei Starovoitov, linux-kernel,
linux-arm-kernel, netdev
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate bitfield instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 16 +++++++++++++
arch/arm64/kernel/insn.c | 56 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 72 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 29386aa..8fd31fc 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -67,6 +67,8 @@ enum aarch64_insn_imm_type {
AARCH64_INSN_IMM_12,
AARCH64_INSN_IMM_9,
AARCH64_INSN_IMM_7,
+ AARCH64_INSN_IMM_S,
+ AARCH64_INSN_IMM_R,
AARCH64_INSN_IMM_MAX
};
@@ -170,6 +172,12 @@ enum aarch64_insn_adsb_type {
AARCH64_INSN_ADSB_SUB_SETFLAGS
};
+enum aarch64_insn_bitfield_type {
+ AARCH64_INSN_BITFIELD_MOVE,
+ AARCH64_INSN_BITFIELD_MOVE_UNSIGNED,
+ AARCH64_INSN_BITFIELD_MOVE_SIGNED
+};
+
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
static __always_inline bool aarch64_insn_is_##abbr(u32 code) \
{ return (code & (mask)) == (val); } \
@@ -186,6 +194,9 @@ __AARCH64_INSN_FUNCS(add_imm, 0x7F000000, 0x11000000)
__AARCH64_INSN_FUNCS(adds_imm, 0x7F000000, 0x31000000)
__AARCH64_INSN_FUNCS(sub_imm, 0x7F000000, 0x51000000)
__AARCH64_INSN_FUNCS(subs_imm, 0x7F000000, 0x71000000)
+__AARCH64_INSN_FUNCS(sbfm, 0x7F800000, 0x13000000)
+__AARCH64_INSN_FUNCS(bfm, 0x7F800000, 0x33000000)
+__AARCH64_INSN_FUNCS(ubfm, 0x7F800000, 0x53000000)
__AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
@@ -236,6 +247,11 @@ u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
enum aarch64_insn_register src,
int imm, enum aarch64_insn_variant variant,
enum aarch64_insn_adsb_type type);
+u32 aarch64_insn_gen_bitfield(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ int immr, int imms,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_bitfield_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index ec3a902..e07d026 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -26,6 +26,7 @@
#include <asm/insn.h>
#define AARCH64_INSN_SF_BIT BIT(31)
+#define AARCH64_INSN_N_BIT BIT(22)
static int aarch64_insn_encoding_class[] = {
AARCH64_INSN_CLS_UNKNOWN,
@@ -259,6 +260,14 @@ u32 __kprobes aarch64_insn_encode_immediate(enum aarch64_insn_imm_type type,
mask = BIT(7) - 1;
shift = 15;
break;
+ case AARCH64_INSN_IMM_S:
+ mask = BIT(6) - 1;
+ shift = 10;
+ break;
+ case AARCH64_INSN_IMM_R:
+ mask = BIT(6) - 1;
+ shift = 16;
+ break;
default:
pr_err("aarch64_insn_encode_immediate: unknown immediate encoding %d\n",
type);
@@ -599,3 +608,50 @@ u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_12, insn, imm);
}
+
+u32 aarch64_insn_gen_bitfield(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ int immr, int imms,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_bitfield_type type)
+{
+ u32 insn;
+ u32 mask;
+
+ switch (type) {
+ case AARCH64_INSN_BITFIELD_MOVE:
+ insn = aarch64_insn_get_bfm_value();
+ break;
+ case AARCH64_INSN_BITFIELD_MOVE_UNSIGNED:
+ insn = aarch64_insn_get_ubfm_value();
+ break;
+ case AARCH64_INSN_BITFIELD_MOVE_SIGNED:
+ insn = aarch64_insn_get_sbfm_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ mask = GENMASK(4, 0);
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ insn |= AARCH64_INSN_SF_BIT | AARCH64_INSN_N_BIT;
+ mask = GENMASK(5, 0);
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ BUG_ON(immr & ~mask);
+ BUG_ON(imms & ~mask);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
+
+ insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_R, insn, immr);
+
+ return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_S, insn, imms);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 08/14] arm64: introduce aarch64_insn_gen_movewide()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, linux-kernel, AKASHI Takahiro, Daniel Borkmann,
netdev, Jiang Liu, David S. Miller, linux-arm-kernel,
Alexei Starovoitov
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate move wide (immediate) instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 13 +++++++++++++
arch/arm64/kernel/insn.c | 43 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 8fd31fc..49dec28 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -172,6 +172,12 @@ enum aarch64_insn_adsb_type {
AARCH64_INSN_ADSB_SUB_SETFLAGS
};
+enum aarch64_insn_movewide_type {
+ AARCH64_INSN_MOVEWIDE_ZERO,
+ AARCH64_INSN_MOVEWIDE_KEEP,
+ AARCH64_INSN_MOVEWIDE_INVERSE
+};
+
enum aarch64_insn_bitfield_type {
AARCH64_INSN_BITFIELD_MOVE,
AARCH64_INSN_BITFIELD_MOVE_UNSIGNED,
@@ -194,9 +200,12 @@ __AARCH64_INSN_FUNCS(add_imm, 0x7F000000, 0x11000000)
__AARCH64_INSN_FUNCS(adds_imm, 0x7F000000, 0x31000000)
__AARCH64_INSN_FUNCS(sub_imm, 0x7F000000, 0x51000000)
__AARCH64_INSN_FUNCS(subs_imm, 0x7F000000, 0x71000000)
+__AARCH64_INSN_FUNCS(movn, 0x7F800000, 0x12800000)
__AARCH64_INSN_FUNCS(sbfm, 0x7F800000, 0x13000000)
__AARCH64_INSN_FUNCS(bfm, 0x7F800000, 0x33000000)
+__AARCH64_INSN_FUNCS(movz, 0x7F800000, 0x52800000)
__AARCH64_INSN_FUNCS(ubfm, 0x7F800000, 0x53000000)
+__AARCH64_INSN_FUNCS(movk, 0x7F800000, 0x72800000)
__AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
@@ -252,6 +261,10 @@ u32 aarch64_insn_gen_bitfield(enum aarch64_insn_register dst,
int immr, int imms,
enum aarch64_insn_variant variant,
enum aarch64_insn_bitfield_type type);
+u32 aarch64_insn_gen_movewide(enum aarch64_insn_register dst,
+ int imm, int shift,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_movewide_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index e07d026..7aa2784 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -655,3 +655,46 @@ u32 aarch64_insn_gen_bitfield(enum aarch64_insn_register dst,
return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_S, insn, imms);
}
+
+u32 aarch64_insn_gen_movewide(enum aarch64_insn_register dst,
+ int imm, int shift,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_movewide_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_MOVEWIDE_ZERO:
+ insn = aarch64_insn_get_movz_value();
+ break;
+ case AARCH64_INSN_MOVEWIDE_KEEP:
+ insn = aarch64_insn_get_movk_value();
+ break;
+ case AARCH64_INSN_MOVEWIDE_INVERSE:
+ insn = aarch64_insn_get_movn_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ BUG_ON(imm & ~(SZ_64K - 1));
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ BUG_ON(shift != 0 && shift != 16);
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ insn |= AARCH64_INSN_SF_BIT;
+ BUG_ON(shift != 0 && shift != 16 && shift != 32 &&
+ shift != 48);
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ insn |= (shift >> 4) << 21;
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
+
+ return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_16, insn, imm);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 09/14] arm64: introduce aarch64_insn_gen_add_sub_shifted_reg()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, Jiang Liu, AKASHI Takahiro, David S. Miller,
Daniel Borkmann, Alexei Starovoitov, linux-kernel,
linux-arm-kernel, netdev
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate add/subtract (shifted register)
instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 11 ++++++++++
arch/arm64/kernel/insn.c | 49 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 60 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 49dec28..c0a765d 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -67,6 +67,7 @@ enum aarch64_insn_imm_type {
AARCH64_INSN_IMM_12,
AARCH64_INSN_IMM_9,
AARCH64_INSN_IMM_7,
+ AARCH64_INSN_IMM_6,
AARCH64_INSN_IMM_S,
AARCH64_INSN_IMM_R,
AARCH64_INSN_IMM_MAX
@@ -206,6 +207,10 @@ __AARCH64_INSN_FUNCS(bfm, 0x7F800000, 0x33000000)
__AARCH64_INSN_FUNCS(movz, 0x7F800000, 0x52800000)
__AARCH64_INSN_FUNCS(ubfm, 0x7F800000, 0x53000000)
__AARCH64_INSN_FUNCS(movk, 0x7F800000, 0x72800000)
+__AARCH64_INSN_FUNCS(add, 0x7F200000, 0x0B000000)
+__AARCH64_INSN_FUNCS(adds, 0x7F200000, 0x2B000000)
+__AARCH64_INSN_FUNCS(sub, 0x7F200000, 0x4B000000)
+__AARCH64_INSN_FUNCS(subs, 0x7F200000, 0x6B000000)
__AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
@@ -265,6 +270,12 @@ u32 aarch64_insn_gen_movewide(enum aarch64_insn_register dst,
int imm, int shift,
enum aarch64_insn_variant variant,
enum aarch64_insn_movewide_type type);
+u32 aarch64_insn_gen_add_sub_shifted_reg(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_register reg,
+ int shift,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_adsb_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 7aa2784..d7a4dd4 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -260,6 +260,7 @@ u32 __kprobes aarch64_insn_encode_immediate(enum aarch64_insn_imm_type type,
mask = BIT(7) - 1;
shift = 15;
break;
+ case AARCH64_INSN_IMM_6:
case AARCH64_INSN_IMM_S:
mask = BIT(6) - 1;
shift = 10;
@@ -698,3 +699,51 @@ u32 aarch64_insn_gen_movewide(enum aarch64_insn_register dst,
return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_16, insn, imm);
}
+
+u32 aarch64_insn_gen_add_sub_shifted_reg(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_register reg,
+ int shift,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_adsb_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_ADSB_ADD:
+ insn = aarch64_insn_get_add_value();
+ break;
+ case AARCH64_INSN_ADSB_SUB:
+ insn = aarch64_insn_get_sub_value();
+ break;
+ case AARCH64_INSN_ADSB_ADD_SETFLAGS:
+ insn = aarch64_insn_get_adds_value();
+ break;
+ case AARCH64_INSN_ADSB_SUB_SETFLAGS:
+ insn = aarch64_insn_get_subs_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ BUG_ON(shift & ~(SZ_32 - 1));
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ insn |= AARCH64_INSN_SF_BIT;
+ BUG_ON(shift & ~(SZ_64 - 1));
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn, reg);
+
+ return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_6, insn, shift);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 10/14] arm64: introduce aarch64_insn_gen_data1()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, linux-kernel, AKASHI Takahiro, Daniel Borkmann,
netdev, Jiang Liu, David S. Miller, linux-arm-kernel,
Alexei Starovoitov
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate data-processing (1 source) instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 13 +++++++++++++
arch/arm64/kernel/insn.c | 37 +++++++++++++++++++++++++++++++++++++
2 files changed, 50 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index c0a765d..246d214 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -185,6 +185,12 @@ enum aarch64_insn_bitfield_type {
AARCH64_INSN_BITFIELD_MOVE_SIGNED
};
+enum aarch64_insn_data1_type {
+ AARCH64_INSN_DATA1_REVERSE_16,
+ AARCH64_INSN_DATA1_REVERSE_32,
+ AARCH64_INSN_DATA1_REVERSE_64,
+};
+
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
static __always_inline bool aarch64_insn_is_##abbr(u32 code) \
{ return (code & (mask)) == (val); } \
@@ -211,6 +217,9 @@ __AARCH64_INSN_FUNCS(add, 0x7F200000, 0x0B000000)
__AARCH64_INSN_FUNCS(adds, 0x7F200000, 0x2B000000)
__AARCH64_INSN_FUNCS(sub, 0x7F200000, 0x4B000000)
__AARCH64_INSN_FUNCS(subs, 0x7F200000, 0x6B000000)
+__AARCH64_INSN_FUNCS(rev16, 0x7FFFFC00, 0x5AC00400)
+__AARCH64_INSN_FUNCS(rev32, 0x7FFFFC00, 0x5AC00800)
+__AARCH64_INSN_FUNCS(rev64, 0x7FFFFC00, 0x5AC00C00)
__AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
@@ -276,6 +285,10 @@ u32 aarch64_insn_gen_add_sub_shifted_reg(enum aarch64_insn_register dst,
int shift,
enum aarch64_insn_variant variant,
enum aarch64_insn_adsb_type type);
+u32 aarch64_insn_gen_data1(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_data1_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index d7a4dd4..81ef3b5 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -747,3 +747,40 @@ u32 aarch64_insn_gen_add_sub_shifted_reg(enum aarch64_insn_register dst,
return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_6, insn, shift);
}
+
+u32 aarch64_insn_gen_data1(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_data1_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_DATA1_REVERSE_16:
+ insn = aarch64_insn_get_rev16_value();
+ break;
+ case AARCH64_INSN_DATA1_REVERSE_32:
+ insn = aarch64_insn_get_rev32_value();
+ break;
+ case AARCH64_INSN_DATA1_REVERSE_64:
+ BUG_ON(variant != AARCH64_INSN_VARIANT_64BIT);
+ insn = aarch64_insn_get_rev64_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ insn |= AARCH64_INSN_SF_BIT;
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
+
+ return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 11/14] arm64: introduce aarch64_insn_gen_data2()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, Jiang Liu, AKASHI Takahiro, David S. Miller,
Daniel Borkmann, Alexei Starovoitov, linux-kernel,
linux-arm-kernel, netdev
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate data-processing (2 source) instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 20 ++++++++++++++++++
arch/arm64/kernel/insn.c | 48 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 68 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 246d214..367245f 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -191,6 +191,15 @@ enum aarch64_insn_data1_type {
AARCH64_INSN_DATA1_REVERSE_64,
};
+enum aarch64_insn_data2_type {
+ AARCH64_INSN_DATA2_UDIV,
+ AARCH64_INSN_DATA2_SDIV,
+ AARCH64_INSN_DATA2_LSLV,
+ AARCH64_INSN_DATA2_LSRV,
+ AARCH64_INSN_DATA2_ASRV,
+ AARCH64_INSN_DATA2_RORV,
+};
+
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
static __always_inline bool aarch64_insn_is_##abbr(u32 code) \
{ return (code & (mask)) == (val); } \
@@ -217,6 +226,12 @@ __AARCH64_INSN_FUNCS(add, 0x7F200000, 0x0B000000)
__AARCH64_INSN_FUNCS(adds, 0x7F200000, 0x2B000000)
__AARCH64_INSN_FUNCS(sub, 0x7F200000, 0x4B000000)
__AARCH64_INSN_FUNCS(subs, 0x7F200000, 0x6B000000)
+__AARCH64_INSN_FUNCS(udiv, 0x7FE0FC00, 0x1AC00800)
+__AARCH64_INSN_FUNCS(sdiv, 0x7FE0FC00, 0x1AC00C00)
+__AARCH64_INSN_FUNCS(lslv, 0x7FE0FC00, 0x1AC02000)
+__AARCH64_INSN_FUNCS(lsrv, 0x7FE0FC00, 0x1AC02400)
+__AARCH64_INSN_FUNCS(asrv, 0x7FE0FC00, 0x1AC02800)
+__AARCH64_INSN_FUNCS(rorv, 0x7FE0FC00, 0x1AC02C00)
__AARCH64_INSN_FUNCS(rev16, 0x7FFFFC00, 0x5AC00400)
__AARCH64_INSN_FUNCS(rev32, 0x7FFFFC00, 0x5AC00800)
__AARCH64_INSN_FUNCS(rev64, 0x7FFFFC00, 0x5AC00C00)
@@ -289,6 +304,11 @@ u32 aarch64_insn_gen_data1(enum aarch64_insn_register dst,
enum aarch64_insn_register src,
enum aarch64_insn_variant variant,
enum aarch64_insn_data1_type type);
+u32 aarch64_insn_gen_data2(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_register reg,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_data2_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 81ef3b5..c054164 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -784,3 +784,51 @@ u32 aarch64_insn_gen_data1(enum aarch64_insn_register dst,
return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
}
+
+u32 aarch64_insn_gen_data2(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_register reg,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_data2_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_DATA2_UDIV:
+ insn = aarch64_insn_get_udiv_value();
+ break;
+ case AARCH64_INSN_DATA2_SDIV:
+ insn = aarch64_insn_get_sdiv_value();
+ break;
+ case AARCH64_INSN_DATA2_LSLV:
+ insn = aarch64_insn_get_lslv_value();
+ break;
+ case AARCH64_INSN_DATA2_LSRV:
+ insn = aarch64_insn_get_lsrv_value();
+ break;
+ case AARCH64_INSN_DATA2_ASRV:
+ insn = aarch64_insn_get_asrv_value();
+ break;
+ case AARCH64_INSN_DATA2_RORV:
+ insn = aarch64_insn_get_rorv_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ insn |= AARCH64_INSN_SF_BIT;
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
+
+ return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn, reg);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 12/14] arm64: introduce aarch64_insn_gen_data3()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, Jiang Liu, AKASHI Takahiro, David S. Miller,
Daniel Borkmann, Alexei Starovoitov, linux-kernel,
linux-arm-kernel, netdev
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate data-processing (3 source) instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 14 ++++++++++++++
arch/arm64/kernel/insn.c | 42 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 367245f..36e8465 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -79,6 +79,7 @@ enum aarch64_insn_register_type {
AARCH64_INSN_REGTYPE_RT2,
AARCH64_INSN_REGTYPE_RM,
AARCH64_INSN_REGTYPE_RD,
+ AARCH64_INSN_REGTYPE_RA,
};
enum aarch64_insn_register {
@@ -200,6 +201,11 @@ enum aarch64_insn_data2_type {
AARCH64_INSN_DATA2_RORV,
};
+enum aarch64_insn_data3_type {
+ AARCH64_INSN_DATA3_MADD,
+ AARCH64_INSN_DATA3_MSUB,
+};
+
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
static __always_inline bool aarch64_insn_is_##abbr(u32 code) \
{ return (code & (mask)) == (val); } \
@@ -226,6 +232,8 @@ __AARCH64_INSN_FUNCS(add, 0x7F200000, 0x0B000000)
__AARCH64_INSN_FUNCS(adds, 0x7F200000, 0x2B000000)
__AARCH64_INSN_FUNCS(sub, 0x7F200000, 0x4B000000)
__AARCH64_INSN_FUNCS(subs, 0x7F200000, 0x6B000000)
+__AARCH64_INSN_FUNCS(madd, 0x7FE08000, 0x1B000000)
+__AARCH64_INSN_FUNCS(msub, 0x7FE08000, 0x1B008000)
__AARCH64_INSN_FUNCS(udiv, 0x7FE0FC00, 0x1AC00800)
__AARCH64_INSN_FUNCS(sdiv, 0x7FE0FC00, 0x1AC00C00)
__AARCH64_INSN_FUNCS(lslv, 0x7FE0FC00, 0x1AC02000)
@@ -309,6 +317,12 @@ u32 aarch64_insn_gen_data2(enum aarch64_insn_register dst,
enum aarch64_insn_register reg,
enum aarch64_insn_variant variant,
enum aarch64_insn_data2_type type);
+u32 aarch64_insn_gen_data3(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_register reg1,
+ enum aarch64_insn_register reg2,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_data3_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index c054164..f73a4bf 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -302,6 +302,7 @@ static u32 aarch64_insn_encode_register(enum aarch64_insn_register_type type,
shift = 5;
break;
case AARCH64_INSN_REGTYPE_RT2:
+ case AARCH64_INSN_REGTYPE_RA:
shift = 10;
break;
case AARCH64_INSN_REGTYPE_RM:
@@ -832,3 +833,44 @@ u32 aarch64_insn_gen_data2(enum aarch64_insn_register dst,
return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn, reg);
}
+
+u32 aarch64_insn_gen_data3(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_register reg1,
+ enum aarch64_insn_register reg2,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_data3_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_DATA3_MADD:
+ insn = aarch64_insn_get_madd_value();
+ break;
+ case AARCH64_INSN_DATA3_MSUB:
+ insn = aarch64_insn_get_msub_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ insn |= AARCH64_INSN_SF_BIT;
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RA, insn, src);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn,
+ reg1);
+
+ return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn,
+ reg2);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 13/14] arm64: introduce aarch64_insn_gen_logical_shifted_reg()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, Jiang Liu, AKASHI Takahiro, David S. Miller,
Daniel Borkmann, Alexei Starovoitov, linux-kernel,
linux-arm-kernel, netdev
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate logical (shifted register)
instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 25 ++++++++++++++++++
arch/arm64/kernel/insn.c | 60 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 85 insertions(+)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 36e8465..56a9e63 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -206,6 +206,17 @@ enum aarch64_insn_data3_type {
AARCH64_INSN_DATA3_MSUB,
};
+enum aarch64_insn_logic_type {
+ AARCH64_INSN_LOGIC_AND,
+ AARCH64_INSN_LOGIC_BIC,
+ AARCH64_INSN_LOGIC_ORR,
+ AARCH64_INSN_LOGIC_ORN,
+ AARCH64_INSN_LOGIC_EOR,
+ AARCH64_INSN_LOGIC_EON,
+ AARCH64_INSN_LOGIC_AND_SETFLAGS,
+ AARCH64_INSN_LOGIC_BIC_SETFLAGS
+};
+
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
static __always_inline bool aarch64_insn_is_##abbr(u32 code) \
{ return (code & (mask)) == (val); } \
@@ -243,6 +254,14 @@ __AARCH64_INSN_FUNCS(rorv, 0x7FE0FC00, 0x1AC02C00)
__AARCH64_INSN_FUNCS(rev16, 0x7FFFFC00, 0x5AC00400)
__AARCH64_INSN_FUNCS(rev32, 0x7FFFFC00, 0x5AC00800)
__AARCH64_INSN_FUNCS(rev64, 0x7FFFFC00, 0x5AC00C00)
+__AARCH64_INSN_FUNCS(and, 0x7F200000, 0x0A000000)
+__AARCH64_INSN_FUNCS(bic, 0x7F200000, 0x0A200000)
+__AARCH64_INSN_FUNCS(orr, 0x7F200000, 0x2A000000)
+__AARCH64_INSN_FUNCS(orn, 0x7F200000, 0x2A200000)
+__AARCH64_INSN_FUNCS(eor, 0x7F200000, 0x4A000000)
+__AARCH64_INSN_FUNCS(eon, 0x7F200000, 0x4A200000)
+__AARCH64_INSN_FUNCS(ands, 0x7F200000, 0x6A000000)
+__AARCH64_INSN_FUNCS(bics, 0x7F200000, 0x6A200000)
__AARCH64_INSN_FUNCS(b, 0xFC000000, 0x14000000)
__AARCH64_INSN_FUNCS(bl, 0xFC000000, 0x94000000)
__AARCH64_INSN_FUNCS(cbz, 0xFE000000, 0x34000000)
@@ -323,6 +342,12 @@ u32 aarch64_insn_gen_data3(enum aarch64_insn_register dst,
enum aarch64_insn_register reg2,
enum aarch64_insn_variant variant,
enum aarch64_insn_data3_type type);
+u32 aarch64_insn_gen_logical_shifted_reg(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_register reg,
+ int shift,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_logic_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index f73a4bf..0668ee5 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -874,3 +874,63 @@ u32 aarch64_insn_gen_data3(enum aarch64_insn_register dst,
return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn,
reg2);
}
+
+u32 aarch64_insn_gen_logical_shifted_reg(enum aarch64_insn_register dst,
+ enum aarch64_insn_register src,
+ enum aarch64_insn_register reg,
+ int shift,
+ enum aarch64_insn_variant variant,
+ enum aarch64_insn_logic_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_LOGIC_AND:
+ insn = aarch64_insn_get_and_value();
+ break;
+ case AARCH64_INSN_LOGIC_BIC:
+ insn = aarch64_insn_get_bic_value();
+ break;
+ case AARCH64_INSN_LOGIC_ORR:
+ insn = aarch64_insn_get_orr_value();
+ break;
+ case AARCH64_INSN_LOGIC_ORN:
+ insn = aarch64_insn_get_orn_value();
+ break;
+ case AARCH64_INSN_LOGIC_EOR:
+ insn = aarch64_insn_get_eor_value();
+ break;
+ case AARCH64_INSN_LOGIC_EON:
+ insn = aarch64_insn_get_eon_value();
+ break;
+ case AARCH64_INSN_LOGIC_AND_SETFLAGS:
+ insn = aarch64_insn_get_ands_value();
+ break;
+ case AARCH64_INSN_LOGIC_BIC_SETFLAGS:
+ insn = aarch64_insn_get_bics_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ switch (variant) {
+ case AARCH64_INSN_VARIANT_32BIT:
+ BUG_ON(shift & ~(SZ_32 - 1));
+ break;
+ case AARCH64_INSN_VARIANT_64BIT:
+ insn |= AARCH64_INSN_SF_BIT;
+ BUG_ON(shift & ~(SZ_64 - 1));
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn, reg);
+
+ return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_6, insn, shift);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 02/14] arm64: introduce aarch64_insn_gen_branch_reg()
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, Jiang Liu, AKASHI Takahiro, David S. Miller,
Daniel Borkmann, Alexei Starovoitov, linux-kernel,
linux-arm-kernel, netdev
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
Introduce function to generate unconditional branch (register)
instructions.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
arch/arm64/include/asm/insn.h | 7 +++++++
arch/arm64/kernel/insn.c | 35 +++++++++++++++++++++++++++++++++--
2 files changed, 40 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index a98c495..5080962 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -71,6 +71,7 @@ enum aarch64_insn_imm_type {
enum aarch64_insn_register_type {
AARCH64_INSN_REGTYPE_RT,
+ AARCH64_INSN_REGTYPE_RN,
};
enum aarch64_insn_register {
@@ -119,6 +120,7 @@ enum aarch64_insn_variant {
enum aarch64_insn_branch_type {
AARCH64_INSN_BRANCH_NOLINK,
AARCH64_INSN_BRANCH_LINK,
+ AARCH64_INSN_BRANCH_RETURN,
AARCH64_INSN_BRANCH_COMP_ZERO,
AARCH64_INSN_BRANCH_COMP_NONZERO,
};
@@ -138,6 +140,9 @@ __AARCH64_INSN_FUNCS(hvc, 0xFFE0001F, 0xD4000002)
__AARCH64_INSN_FUNCS(smc, 0xFFE0001F, 0xD4000003)
__AARCH64_INSN_FUNCS(brk, 0xFFE0001F, 0xD4200000)
__AARCH64_INSN_FUNCS(hint, 0xFFFFF01F, 0xD503201F)
+__AARCH64_INSN_FUNCS(br, 0xFFFFFC1F, 0xD61F0000)
+__AARCH64_INSN_FUNCS(blr, 0xFFFFFC1F, 0xD63F0000)
+__AARCH64_INSN_FUNCS(ret, 0xFFFFFC1F, 0xD65F0000)
#undef __AARCH64_INSN_FUNCS
@@ -156,6 +161,8 @@ u32 aarch64_insn_gen_comp_branch_imm(unsigned long pc, unsigned long addr,
enum aarch64_insn_branch_type type);
u32 aarch64_insn_gen_hint(enum aarch64_insn_hint_op op);
u32 aarch64_insn_gen_nop(void);
+u32 aarch64_insn_gen_branch_reg(enum aarch64_insn_register reg,
+ enum aarch64_insn_branch_type type);
bool aarch64_insn_hotpatch_safe(u32 old_insn, u32 new_insn);
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index d9f7827..6797936 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -283,6 +283,9 @@ static u32 aarch64_insn_encode_register(enum aarch64_insn_register_type type,
case AARCH64_INSN_REGTYPE_RT:
shift = 0;
break;
+ case AARCH64_INSN_REGTYPE_RN:
+ shift = 5;
+ break;
default:
pr_err("%s: unknown register type encoding %d\n", __func__,
type);
@@ -325,10 +328,16 @@ u32 __kprobes aarch64_insn_gen_branch_imm(unsigned long pc, unsigned long addr,
*/
offset = branch_imm_common(pc, addr, SZ_128M);
- if (type == AARCH64_INSN_BRANCH_LINK)
+ switch (type) {
+ case AARCH64_INSN_BRANCH_LINK:
insn = aarch64_insn_get_bl_value();
- else
+ break;
+ case AARCH64_INSN_BRANCH_NOLINK:
insn = aarch64_insn_get_b_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_26, insn,
offset >> 2);
@@ -380,3 +389,25 @@ u32 __kprobes aarch64_insn_gen_nop(void)
{
return aarch64_insn_gen_hint(AARCH64_INSN_HINT_NOP);
}
+
+u32 aarch64_insn_gen_branch_reg(enum aarch64_insn_register reg,
+ enum aarch64_insn_branch_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_BRANCH_NOLINK:
+ insn = aarch64_insn_get_br_value();
+ break;
+ case AARCH64_INSN_BRANCH_LINK:
+ insn = aarch64_insn_get_blr_value();
+ break;
+ case AARCH64_INSN_BRANCH_RETURN:
+ insn = aarch64_insn_get_ret_value();
+ break;
+ default:
+ BUG_ON(1);
+ }
+
+ return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, reg);
+}
--
1.9.1
^ permalink raw reply related
* [PATCHv2 14/14] arm64: eBPF JIT compiler
From: Zi Shen Lim @ 2014-08-27 4:15 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon
Cc: Zi Shen Lim, David S. Miller, Daniel Borkmann, Alexei Starovoitov,
Chema Gonzalez, linux-kernel, linux-arm-kernel, netdev
In-Reply-To: <1409112930-25677-1-git-send-email-zlim.lnx@gmail.com>
The JIT compiler emits A64 instructions. It supports eBPF only.
Legacy BPF is supported thanks to conversion by BPF core.
JIT is enabled in the same way as for other architectures:
echo 1 > /proc/sys/net/core/bpf_jit_enable
Or for additional compiler output:
echo 2 > /proc/sys/net/core/bpf_jit_enable
See Documentation/networking/filter.txt for more information.
The implementation passes all 57 tests in lib/test_bpf.c
on ARMv8 Foundation Model :) Also tested by Will on Juno platform.
Signed-off-by: Zi Shen Lim <zlim.lnx@gmail.com>
Acked-by: Alexei Starovoitov <ast@plumgrid.com>
Acked-by: Will Deacon <will.deacon@arm.com>
---
v1->v2:
Rebased onto 3.17-rc2, and fixed up changes related to:
- sock_filter_int -> bpf_insn: 2695fb552cbe (net: filter: rename
'struct sock_filter_int' into 'struct bpf_insn')
- sk_filter -> bpf_prog: 7ae457c1e5b4 (net: filter: split
'struct sk_filter' into socket and bpf parts)
RFCv3->v1:
Addressed review comments from Will wrt codegen bits:
- define and use {SF,N}_BIT
- use masks for limit checks
Also:
- rebase onto net-next
RFCv2->RFCv3:
- clarify 16B stack alignment requirement - I missed one reference
- fixed a couple checks for immediate bits
- make bpf_jit.h checkpatch clean
- remove stale DW case in LD_IND and LD_ABS (good catch by Alexei)
- add Alexei's Acked-by
- rebase onto net-next
Also, per discussion with Will, consolidated bpf_jit.h into
arch/arm64/.../insn.{c,h}:
- instruction encoding stuff moved into arch/arm64/kernel/insn.c
- bpf_jit.h uses arch/arm64/include/asm/insn.h
RFCv1->RFCv2:
Addressed review comments from Alexei:
- use core-$(CONFIG_NET)
- use GENMASK
- lower-case function names in header file
- drop LD_ABS+DW and LD_IND+DW, which do not exist in eBPF yet
- use pr_xxx_once() to prevent spamming logs
- clarify 16B stack alignment requirement
- drop usage of EMIT macro which was saving just one argument,
turns out having additional argument wasn't too much of an eyesore
Also, per discussion with Alexei, and additional suggestion from
Daniel:
- moved load_pointer() from net/core/filter.c into filter.h
as bpf_load_pointer()
which is done as a separate preparatory patch. [1]
[1] http://patchwork.ozlabs.org/patch/366906/
NOTES:
* The preparatory patch [1] has been merged into net-next
9f12fbe603f7 ("net: filter: move load_pointer() into filter.h").
* bpf_jit_comp.c and bpf_jit.h is checkpatch clean.
* The following sparse warning is not applicable:
warning: symbol 'bpf_jit_enable' was not declared. Should it be static?
FUTURE WORK:
1. Implement remaining classes of eBPF instructions: ST|MEM, STX|XADD
which currently do not have corresponding test cases in test_bpf.
2. Further compiler optimization, such as optimization for small
immediates.
Documentation/networking/filter.txt | 6 +-
arch/arm64/Kconfig | 1 +
arch/arm64/Makefile | 1 +
arch/arm64/net/Makefile | 4 +
arch/arm64/net/bpf_jit.h | 169 +++++++++
arch/arm64/net/bpf_jit_comp.c | 677 ++++++++++++++++++++++++++++++++++++
6 files changed, 855 insertions(+), 3 deletions(-)
create mode 100644 arch/arm64/net/Makefile
create mode 100644 arch/arm64/net/bpf_jit.h
create mode 100644 arch/arm64/net/bpf_jit_comp.c
diff --git a/Documentation/networking/filter.txt b/Documentation/networking/filter.txt
index c48a970..1842d4f 100644
--- a/Documentation/networking/filter.txt
+++ b/Documentation/networking/filter.txt
@@ -462,9 +462,9 @@ JIT compiler
------------
The Linux kernel has a built-in BPF JIT compiler for x86_64, SPARC, PowerPC,
-ARM and s390 and can be enabled through CONFIG_BPF_JIT. The JIT compiler is
-transparently invoked for each attached filter from user space or for internal
-kernel users if it has been previously enabled by root:
+ARM, ARM64 and s390 and can be enabled through CONFIG_BPF_JIT. The JIT compiler
+is transparently invoked for each attached filter from user space or for
+internal kernel users if it has been previously enabled by root:
echo 1 > /proc/sys/net/core/bpf_jit_enable
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fd4e81a..cfea623 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -35,6 +35,7 @@ config ARM64
select HAVE_ARCH_JUMP_LABEL
select HAVE_ARCH_KGDB
select HAVE_ARCH_TRACEHOOK
+ select HAVE_BPF_JIT
select HAVE_C_RECORDMCOUNT
select HAVE_CC_STACKPROTECTOR
select HAVE_DEBUG_BUGVERBOSE
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 2df5e5d..59c86b6 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -47,6 +47,7 @@ endif
export TEXT_OFFSET GZFLAGS
core-y += arch/arm64/kernel/ arch/arm64/mm/
+core-$(CONFIG_NET) += arch/arm64/net/
core-$(CONFIG_KVM) += arch/arm64/kvm/
core-$(CONFIG_XEN) += arch/arm64/xen/
core-$(CONFIG_CRYPTO) += arch/arm64/crypto/
diff --git a/arch/arm64/net/Makefile b/arch/arm64/net/Makefile
new file mode 100644
index 0000000..da97633
--- /dev/null
+++ b/arch/arm64/net/Makefile
@@ -0,0 +1,4 @@
+#
+# ARM64 networking code
+#
+obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o
diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h
new file mode 100644
index 0000000..2134f7e
--- /dev/null
+++ b/arch/arm64/net/bpf_jit.h
@@ -0,0 +1,169 @@
+/*
+ * BPF JIT compiler for ARM64
+ *
+ * Copyright (C) 2014 Zi Shen Lim <zlim.lnx@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef _BPF_JIT_H
+#define _BPF_JIT_H
+
+#include <asm/insn.h>
+
+/* 5-bit Register Operand */
+#define A64_R(x) AARCH64_INSN_REG_##x
+#define A64_FP AARCH64_INSN_REG_FP
+#define A64_LR AARCH64_INSN_REG_LR
+#define A64_ZR AARCH64_INSN_REG_ZR
+#define A64_SP AARCH64_INSN_REG_SP
+
+#define A64_VARIANT(sf) \
+ ((sf) ? AARCH64_INSN_VARIANT_64BIT : AARCH64_INSN_VARIANT_32BIT)
+
+/* Compare & branch (immediate) */
+#define A64_COMP_BRANCH(sf, Rt, offset, type) \
+ aarch64_insn_gen_comp_branch_imm(0, offset, Rt, A64_VARIANT(sf), \
+ AARCH64_INSN_BRANCH_COMP_##type)
+#define A64_CBZ(sf, Rt, imm19) A64_COMP_BRANCH(sf, Rt, (imm19) << 2, ZERO)
+
+/* Conditional branch (immediate) */
+#define A64_COND_BRANCH(cond, offset) \
+ aarch64_insn_gen_cond_branch_imm(0, offset, cond)
+#define A64_COND_EQ AARCH64_INSN_COND_EQ /* == */
+#define A64_COND_NE AARCH64_INSN_COND_NE /* != */
+#define A64_COND_CS AARCH64_INSN_COND_CS /* unsigned >= */
+#define A64_COND_HI AARCH64_INSN_COND_HI /* unsigned > */
+#define A64_COND_GE AARCH64_INSN_COND_GE /* signed >= */
+#define A64_COND_GT AARCH64_INSN_COND_GT /* signed > */
+#define A64_B_(cond, imm19) A64_COND_BRANCH(cond, (imm19) << 2)
+
+/* Unconditional branch (immediate) */
+#define A64_BRANCH(offset, type) aarch64_insn_gen_branch_imm(0, offset, \
+ AARCH64_INSN_BRANCH_##type)
+#define A64_B(imm26) A64_BRANCH((imm26) << 2, NOLINK)
+#define A64_BL(imm26) A64_BRANCH((imm26) << 2, LINK)
+
+/* Unconditional branch (register) */
+#define A64_BLR(Rn) aarch64_insn_gen_branch_reg(Rn, AARCH64_INSN_BRANCH_LINK)
+#define A64_RET(Rn) aarch64_insn_gen_branch_reg(Rn, AARCH64_INSN_BRANCH_RETURN)
+
+/* Load/store register (register offset) */
+#define A64_LS_REG(Rt, Rn, Rm, size, type) \
+ aarch64_insn_gen_load_store_reg(Rt, Rn, Rm, \
+ AARCH64_INSN_SIZE_##size, \
+ AARCH64_INSN_LDST_##type##_REG_OFFSET)
+#define A64_STRB(Wt, Xn, Xm) A64_LS_REG(Wt, Xn, Xm, 8, STORE)
+#define A64_LDRB(Wt, Xn, Xm) A64_LS_REG(Wt, Xn, Xm, 8, LOAD)
+#define A64_STRH(Wt, Xn, Xm) A64_LS_REG(Wt, Xn, Xm, 16, STORE)
+#define A64_LDRH(Wt, Xn, Xm) A64_LS_REG(Wt, Xn, Xm, 16, LOAD)
+#define A64_STR32(Wt, Xn, Xm) A64_LS_REG(Wt, Xn, Xm, 32, STORE)
+#define A64_LDR32(Wt, Xn, Xm) A64_LS_REG(Wt, Xn, Xm, 32, LOAD)
+#define A64_STR64(Xt, Xn, Xm) A64_LS_REG(Xt, Xn, Xm, 64, STORE)
+#define A64_LDR64(Xt, Xn, Xm) A64_LS_REG(Xt, Xn, Xm, 64, LOAD)
+
+/* Load/store register pair */
+#define A64_LS_PAIR(Rt, Rt2, Rn, offset, ls, type) \
+ aarch64_insn_gen_load_store_pair(Rt, Rt2, Rn, offset, \
+ AARCH64_INSN_VARIANT_64BIT, \
+ AARCH64_INSN_LDST_##ls##_PAIR_##type)
+/* Rn -= 16; Rn[0] = Rt; Rn[8] = Rt2; */
+#define A64_PUSH(Rt, Rt2, Rn) A64_LS_PAIR(Rt, Rt2, Rn, -16, STORE, PRE_INDEX)
+/* Rt = Rn[0]; Rt2 = Rn[8]; Rn += 16; */
+#define A64_POP(Rt, Rt2, Rn) A64_LS_PAIR(Rt, Rt2, Rn, 16, LOAD, POST_INDEX)
+
+/* Add/subtract (immediate) */
+#define A64_ADDSUB_IMM(sf, Rd, Rn, imm12, type) \
+ aarch64_insn_gen_add_sub_imm(Rd, Rn, imm12, \
+ A64_VARIANT(sf), AARCH64_INSN_ADSB_##type)
+/* Rd = Rn OP imm12 */
+#define A64_ADD_I(sf, Rd, Rn, imm12) A64_ADDSUB_IMM(sf, Rd, Rn, imm12, ADD)
+#define A64_SUB_I(sf, Rd, Rn, imm12) A64_ADDSUB_IMM(sf, Rd, Rn, imm12, SUB)
+/* Rd = Rn */
+#define A64_MOV(sf, Rd, Rn) A64_ADD_I(sf, Rd, Rn, 0)
+
+/* Bitfield move */
+#define A64_BITFIELD(sf, Rd, Rn, immr, imms, type) \
+ aarch64_insn_gen_bitfield(Rd, Rn, immr, imms, \
+ A64_VARIANT(sf), AARCH64_INSN_BITFIELD_MOVE_##type)
+/* Signed, with sign replication to left and zeros to right */
+#define A64_SBFM(sf, Rd, Rn, ir, is) A64_BITFIELD(sf, Rd, Rn, ir, is, SIGNED)
+/* Unsigned, with zeros to left and right */
+#define A64_UBFM(sf, Rd, Rn, ir, is) A64_BITFIELD(sf, Rd, Rn, ir, is, UNSIGNED)
+
+/* Rd = Rn << shift */
+#define A64_LSL(sf, Rd, Rn, shift) ({ \
+ int sz = (sf) ? 64 : 32; \
+ A64_UBFM(sf, Rd, Rn, (unsigned)-(shift) % sz, sz - 1 - (shift)); \
+})
+/* Rd = Rn >> shift */
+#define A64_LSR(sf, Rd, Rn, shift) A64_UBFM(sf, Rd, Rn, shift, (sf) ? 63 : 31)
+/* Rd = Rn >> shift; signed */
+#define A64_ASR(sf, Rd, Rn, shift) A64_SBFM(sf, Rd, Rn, shift, (sf) ? 63 : 31)
+
+/* Move wide (immediate) */
+#define A64_MOVEW(sf, Rd, imm16, shift, type) \
+ aarch64_insn_gen_movewide(Rd, imm16, shift, \
+ A64_VARIANT(sf), AARCH64_INSN_MOVEWIDE_##type)
+/* Rd = Zeros (for MOVZ);
+ * Rd |= imm16 << shift (where shift is {0, 16, 32, 48});
+ * Rd = ~Rd; (for MOVN); */
+#define A64_MOVN(sf, Rd, imm16, shift) A64_MOVEW(sf, Rd, imm16, shift, INVERSE)
+#define A64_MOVZ(sf, Rd, imm16, shift) A64_MOVEW(sf, Rd, imm16, shift, ZERO)
+#define A64_MOVK(sf, Rd, imm16, shift) A64_MOVEW(sf, Rd, imm16, shift, KEEP)
+
+/* Add/subtract (shifted register) */
+#define A64_ADDSUB_SREG(sf, Rd, Rn, Rm, type) \
+ aarch64_insn_gen_add_sub_shifted_reg(Rd, Rn, Rm, 0, \
+ A64_VARIANT(sf), AARCH64_INSN_ADSB_##type)
+/* Rd = Rn OP Rm */
+#define A64_ADD(sf, Rd, Rn, Rm) A64_ADDSUB_SREG(sf, Rd, Rn, Rm, ADD)
+#define A64_SUB(sf, Rd, Rn, Rm) A64_ADDSUB_SREG(sf, Rd, Rn, Rm, SUB)
+#define A64_SUBS(sf, Rd, Rn, Rm) A64_ADDSUB_SREG(sf, Rd, Rn, Rm, SUB_SETFLAGS)
+/* Rd = -Rm */
+#define A64_NEG(sf, Rd, Rm) A64_SUB(sf, Rd, A64_ZR, Rm)
+/* Rn - Rm; set condition flags */
+#define A64_CMP(sf, Rn, Rm) A64_SUBS(sf, A64_ZR, Rn, Rm)
+
+/* Data-processing (1 source) */
+#define A64_DATA1(sf, Rd, Rn, type) aarch64_insn_gen_data1(Rd, Rn, \
+ A64_VARIANT(sf), AARCH64_INSN_DATA1_##type)
+/* Rd = BSWAPx(Rn) */
+#define A64_REV16(sf, Rd, Rn) A64_DATA1(sf, Rd, Rn, REVERSE_16)
+#define A64_REV32(sf, Rd, Rn) A64_DATA1(sf, Rd, Rn, REVERSE_32)
+#define A64_REV64(Rd, Rn) A64_DATA1(1, Rd, Rn, REVERSE_64)
+
+/* Data-processing (2 source) */
+/* Rd = Rn OP Rm */
+#define A64_UDIV(sf, Rd, Rn, Rm) aarch64_insn_gen_data2(Rd, Rn, Rm, \
+ A64_VARIANT(sf), AARCH64_INSN_DATA2_UDIV)
+
+/* Data-processing (3 source) */
+/* Rd = Ra + Rn * Rm */
+#define A64_MADD(sf, Rd, Ra, Rn, Rm) aarch64_insn_gen_data3(Rd, Ra, Rn, Rm, \
+ A64_VARIANT(sf), AARCH64_INSN_DATA3_MADD)
+/* Rd = Rn * Rm */
+#define A64_MUL(sf, Rd, Rn, Rm) A64_MADD(sf, Rd, A64_ZR, Rn, Rm)
+
+/* Logical (shifted register) */
+#define A64_LOGIC_SREG(sf, Rd, Rn, Rm, type) \
+ aarch64_insn_gen_logical_shifted_reg(Rd, Rn, Rm, 0, \
+ A64_VARIANT(sf), AARCH64_INSN_LOGIC_##type)
+/* Rd = Rn OP Rm */
+#define A64_AND(sf, Rd, Rn, Rm) A64_LOGIC_SREG(sf, Rd, Rn, Rm, AND)
+#define A64_ORR(sf, Rd, Rn, Rm) A64_LOGIC_SREG(sf, Rd, Rn, Rm, ORR)
+#define A64_EOR(sf, Rd, Rn, Rm) A64_LOGIC_SREG(sf, Rd, Rn, Rm, EOR)
+#define A64_ANDS(sf, Rd, Rn, Rm) A64_LOGIC_SREG(sf, Rd, Rn, Rm, AND_SETFLAGS)
+/* Rn & Rm; set condition flags */
+#define A64_TST(sf, Rn, Rm) A64_ANDS(sf, A64_ZR, Rn, Rm)
+
+#endif /* _BPF_JIT_H */
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
new file mode 100644
index 0000000..38c4296
--- /dev/null
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -0,0 +1,677 @@
+/*
+ * BPF JIT compiler for ARM64
+ *
+ * Copyright (C) 2014 Zi Shen Lim <zlim.lnx@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#define pr_fmt(fmt) "bpf_jit: " fmt
+
+#include <linux/filter.h>
+#include <linux/moduleloader.h>
+#include <linux/printk.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <asm/byteorder.h>
+#include <asm/cacheflush.h>
+
+#include "bpf_jit.h"
+
+int bpf_jit_enable __read_mostly;
+
+#define TMP_REG_1 (MAX_BPF_REG + 0)
+#define TMP_REG_2 (MAX_BPF_REG + 1)
+
+/* Map BPF registers to A64 registers */
+static const int bpf2a64[] = {
+ /* return value from in-kernel function, and exit value from eBPF */
+ [BPF_REG_0] = A64_R(7),
+ /* arguments from eBPF program to in-kernel function */
+ [BPF_REG_1] = A64_R(0),
+ [BPF_REG_2] = A64_R(1),
+ [BPF_REG_3] = A64_R(2),
+ [BPF_REG_4] = A64_R(3),
+ [BPF_REG_5] = A64_R(4),
+ /* callee saved registers that in-kernel function will preserve */
+ [BPF_REG_6] = A64_R(19),
+ [BPF_REG_7] = A64_R(20),
+ [BPF_REG_8] = A64_R(21),
+ [BPF_REG_9] = A64_R(22),
+ /* read-only frame pointer to access stack */
+ [BPF_REG_FP] = A64_FP,
+ /* temporary register for internal BPF JIT */
+ [TMP_REG_1] = A64_R(23),
+ [TMP_REG_2] = A64_R(24),
+};
+
+struct jit_ctx {
+ const struct bpf_prog *prog;
+ int idx;
+ int tmp_used;
+ int body_offset;
+ int *offset;
+ u32 *image;
+};
+
+static inline void emit(const u32 insn, struct jit_ctx *ctx)
+{
+ if (ctx->image != NULL)
+ ctx->image[ctx->idx] = cpu_to_le32(insn);
+
+ ctx->idx++;
+}
+
+static inline void emit_a64_mov_i64(const int reg, const u64 val,
+ struct jit_ctx *ctx)
+{
+ u64 tmp = val;
+ int shift = 0;
+
+ emit(A64_MOVZ(1, reg, tmp & 0xffff, shift), ctx);
+ tmp >>= 16;
+ shift += 16;
+ while (tmp) {
+ if (tmp & 0xffff)
+ emit(A64_MOVK(1, reg, tmp & 0xffff, shift), ctx);
+ tmp >>= 16;
+ shift += 16;
+ }
+}
+
+static inline void emit_a64_mov_i(const int is64, const int reg,
+ const s32 val, struct jit_ctx *ctx)
+{
+ u16 hi = val >> 16;
+ u16 lo = val & 0xffff;
+
+ if (hi & 0x8000) {
+ if (hi == 0xffff) {
+ emit(A64_MOVN(is64, reg, (u16)~lo, 0), ctx);
+ } else {
+ emit(A64_MOVN(is64, reg, (u16)~hi, 16), ctx);
+ emit(A64_MOVK(is64, reg, lo, 0), ctx);
+ }
+ } else {
+ emit(A64_MOVZ(is64, reg, lo, 0), ctx);
+ if (hi)
+ emit(A64_MOVK(is64, reg, hi, 16), ctx);
+ }
+}
+
+static inline int bpf2a64_offset(int bpf_to, int bpf_from,
+ const struct jit_ctx *ctx)
+{
+ int to = ctx->offset[bpf_to + 1];
+ /* -1 to account for the Branch instruction */
+ int from = ctx->offset[bpf_from + 1] - 1;
+
+ return to - from;
+}
+
+static inline int epilogue_offset(const struct jit_ctx *ctx)
+{
+ int to = ctx->offset[ctx->prog->len - 1];
+ int from = ctx->idx - ctx->body_offset;
+
+ return to - from;
+}
+
+/* Stack must be multiples of 16B */
+#define STACK_ALIGN(sz) (((sz) + 15) & ~15)
+
+static void build_prologue(struct jit_ctx *ctx)
+{
+ const u8 r6 = bpf2a64[BPF_REG_6];
+ const u8 r7 = bpf2a64[BPF_REG_7];
+ const u8 r8 = bpf2a64[BPF_REG_8];
+ const u8 r9 = bpf2a64[BPF_REG_9];
+ const u8 fp = bpf2a64[BPF_REG_FP];
+ const u8 ra = bpf2a64[BPF_REG_A];
+ const u8 rx = bpf2a64[BPF_REG_X];
+ const u8 tmp1 = bpf2a64[TMP_REG_1];
+ const u8 tmp2 = bpf2a64[TMP_REG_2];
+ int stack_size = MAX_BPF_STACK;
+
+ stack_size += 4; /* extra for skb_copy_bits buffer */
+ stack_size = STACK_ALIGN(stack_size);
+
+ /* Save callee-saved register */
+ emit(A64_PUSH(r6, r7, A64_SP), ctx);
+ emit(A64_PUSH(r8, r9, A64_SP), ctx);
+ if (ctx->tmp_used)
+ emit(A64_PUSH(tmp1, tmp2, A64_SP), ctx);
+
+ /* Set up BPF stack */
+ emit(A64_SUB_I(1, A64_SP, A64_SP, stack_size), ctx);
+
+ /* Set up frame pointer */
+ emit(A64_MOV(1, fp, A64_SP), ctx);
+
+ /* Clear registers A and X */
+ emit_a64_mov_i64(ra, 0, ctx);
+ emit_a64_mov_i64(rx, 0, ctx);
+}
+
+static void build_epilogue(struct jit_ctx *ctx)
+{
+ const u8 r0 = bpf2a64[BPF_REG_0];
+ const u8 r6 = bpf2a64[BPF_REG_6];
+ const u8 r7 = bpf2a64[BPF_REG_7];
+ const u8 r8 = bpf2a64[BPF_REG_8];
+ const u8 r9 = bpf2a64[BPF_REG_9];
+ const u8 fp = bpf2a64[BPF_REG_FP];
+ const u8 tmp1 = bpf2a64[TMP_REG_1];
+ const u8 tmp2 = bpf2a64[TMP_REG_2];
+ int stack_size = MAX_BPF_STACK;
+
+ stack_size += 4; /* extra for skb_copy_bits buffer */
+ stack_size = STACK_ALIGN(stack_size);
+
+ /* We're done with BPF stack */
+ emit(A64_ADD_I(1, A64_SP, A64_SP, stack_size), ctx);
+
+ /* Restore callee-saved register */
+ if (ctx->tmp_used)
+ emit(A64_POP(tmp1, tmp2, A64_SP), ctx);
+ emit(A64_POP(r8, r9, A64_SP), ctx);
+ emit(A64_POP(r6, r7, A64_SP), ctx);
+
+ /* Restore frame pointer */
+ emit(A64_MOV(1, fp, A64_SP), ctx);
+
+ /* Set return value */
+ emit(A64_MOV(1, A64_R(0), r0), ctx);
+
+ emit(A64_RET(A64_LR), ctx);
+}
+
+static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx)
+{
+ const u8 code = insn->code;
+ const u8 dst = bpf2a64[insn->dst_reg];
+ const u8 src = bpf2a64[insn->src_reg];
+ const u8 tmp = bpf2a64[TMP_REG_1];
+ const u8 tmp2 = bpf2a64[TMP_REG_2];
+ const s16 off = insn->off;
+ const s32 imm = insn->imm;
+ const int i = insn - ctx->prog->insnsi;
+ const bool is64 = BPF_CLASS(code) == BPF_ALU64;
+ u8 jmp_cond;
+ s32 jmp_offset;
+
+ switch (code) {
+ /* dst = src */
+ case BPF_ALU | BPF_MOV | BPF_X:
+ case BPF_ALU64 | BPF_MOV | BPF_X:
+ emit(A64_MOV(is64, dst, src), ctx);
+ break;
+ /* dst = dst OP src */
+ case BPF_ALU | BPF_ADD | BPF_X:
+ case BPF_ALU64 | BPF_ADD | BPF_X:
+ emit(A64_ADD(is64, dst, dst, src), ctx);
+ break;
+ case BPF_ALU | BPF_SUB | BPF_X:
+ case BPF_ALU64 | BPF_SUB | BPF_X:
+ emit(A64_SUB(is64, dst, dst, src), ctx);
+ break;
+ case BPF_ALU | BPF_AND | BPF_X:
+ case BPF_ALU64 | BPF_AND | BPF_X:
+ emit(A64_AND(is64, dst, dst, src), ctx);
+ break;
+ case BPF_ALU | BPF_OR | BPF_X:
+ case BPF_ALU64 | BPF_OR | BPF_X:
+ emit(A64_ORR(is64, dst, dst, src), ctx);
+ break;
+ case BPF_ALU | BPF_XOR | BPF_X:
+ case BPF_ALU64 | BPF_XOR | BPF_X:
+ emit(A64_EOR(is64, dst, dst, src), ctx);
+ break;
+ case BPF_ALU | BPF_MUL | BPF_X:
+ case BPF_ALU64 | BPF_MUL | BPF_X:
+ emit(A64_MUL(is64, dst, dst, src), ctx);
+ break;
+ case BPF_ALU | BPF_DIV | BPF_X:
+ case BPF_ALU64 | BPF_DIV | BPF_X:
+ emit(A64_UDIV(is64, dst, dst, src), ctx);
+ break;
+ case BPF_ALU | BPF_MOD | BPF_X:
+ case BPF_ALU64 | BPF_MOD | BPF_X:
+ ctx->tmp_used = 1;
+ emit(A64_UDIV(is64, tmp, dst, src), ctx);
+ emit(A64_MUL(is64, tmp, tmp, src), ctx);
+ emit(A64_SUB(is64, dst, dst, tmp), ctx);
+ break;
+ /* dst = -dst */
+ case BPF_ALU | BPF_NEG:
+ case BPF_ALU64 | BPF_NEG:
+ emit(A64_NEG(is64, dst, dst), ctx);
+ break;
+ /* dst = BSWAP##imm(dst) */
+ case BPF_ALU | BPF_END | BPF_FROM_LE:
+ case BPF_ALU | BPF_END | BPF_FROM_BE:
+#ifdef CONFIG_CPU_BIG_ENDIAN
+ if (BPF_SRC(code) == BPF_FROM_BE)
+ break;
+#else /* !CONFIG_CPU_BIG_ENDIAN */
+ if (BPF_SRC(code) == BPF_FROM_LE)
+ break;
+#endif
+ switch (imm) {
+ case 16:
+ emit(A64_REV16(is64, dst, dst), ctx);
+ break;
+ case 32:
+ emit(A64_REV32(is64, dst, dst), ctx);
+ break;
+ case 64:
+ emit(A64_REV64(dst, dst), ctx);
+ break;
+ }
+ break;
+ /* dst = imm */
+ case BPF_ALU | BPF_MOV | BPF_K:
+ case BPF_ALU64 | BPF_MOV | BPF_K:
+ emit_a64_mov_i(is64, dst, imm, ctx);
+ break;
+ /* dst = dst OP imm */
+ case BPF_ALU | BPF_ADD | BPF_K:
+ case BPF_ALU64 | BPF_ADD | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(is64, tmp, imm, ctx);
+ emit(A64_ADD(is64, dst, dst, tmp), ctx);
+ break;
+ case BPF_ALU | BPF_SUB | BPF_K:
+ case BPF_ALU64 | BPF_SUB | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(is64, tmp, imm, ctx);
+ emit(A64_SUB(is64, dst, dst, tmp), ctx);
+ break;
+ case BPF_ALU | BPF_AND | BPF_K:
+ case BPF_ALU64 | BPF_AND | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(is64, tmp, imm, ctx);
+ emit(A64_AND(is64, dst, dst, tmp), ctx);
+ break;
+ case BPF_ALU | BPF_OR | BPF_K:
+ case BPF_ALU64 | BPF_OR | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(is64, tmp, imm, ctx);
+ emit(A64_ORR(is64, dst, dst, tmp), ctx);
+ break;
+ case BPF_ALU | BPF_XOR | BPF_K:
+ case BPF_ALU64 | BPF_XOR | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(is64, tmp, imm, ctx);
+ emit(A64_EOR(is64, dst, dst, tmp), ctx);
+ break;
+ case BPF_ALU | BPF_MUL | BPF_K:
+ case BPF_ALU64 | BPF_MUL | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(is64, tmp, imm, ctx);
+ emit(A64_MUL(is64, dst, dst, tmp), ctx);
+ break;
+ case BPF_ALU | BPF_DIV | BPF_K:
+ case BPF_ALU64 | BPF_DIV | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(is64, tmp, imm, ctx);
+ emit(A64_UDIV(is64, dst, dst, tmp), ctx);
+ break;
+ case BPF_ALU | BPF_MOD | BPF_K:
+ case BPF_ALU64 | BPF_MOD | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(is64, tmp2, imm, ctx);
+ emit(A64_UDIV(is64, tmp, dst, tmp2), ctx);
+ emit(A64_MUL(is64, tmp, tmp, tmp2), ctx);
+ emit(A64_SUB(is64, dst, dst, tmp), ctx);
+ break;
+ case BPF_ALU | BPF_LSH | BPF_K:
+ case BPF_ALU64 | BPF_LSH | BPF_K:
+ emit(A64_LSL(is64, dst, dst, imm), ctx);
+ break;
+ case BPF_ALU | BPF_RSH | BPF_K:
+ case BPF_ALU64 | BPF_RSH | BPF_K:
+ emit(A64_LSR(is64, dst, dst, imm), ctx);
+ break;
+ case BPF_ALU | BPF_ARSH | BPF_K:
+ case BPF_ALU64 | BPF_ARSH | BPF_K:
+ emit(A64_ASR(is64, dst, dst, imm), ctx);
+ break;
+
+#define check_imm(bits, imm) do { \
+ if ((((imm) > 0) && ((imm) >> (bits))) || \
+ (((imm) < 0) && (~(imm) >> (bits)))) { \
+ pr_info("[%2d] imm=%d(0x%x) out of range\n", \
+ i, imm, imm); \
+ return -EINVAL; \
+ } \
+} while (0)
+#define check_imm19(imm) check_imm(19, imm)
+#define check_imm26(imm) check_imm(26, imm)
+
+ /* JUMP off */
+ case BPF_JMP | BPF_JA:
+ jmp_offset = bpf2a64_offset(i + off, i, ctx);
+ check_imm26(jmp_offset);
+ emit(A64_B(jmp_offset), ctx);
+ break;
+ /* IF (dst COND src) JUMP off */
+ case BPF_JMP | BPF_JEQ | BPF_X:
+ case BPF_JMP | BPF_JGT | BPF_X:
+ case BPF_JMP | BPF_JGE | BPF_X:
+ case BPF_JMP | BPF_JNE | BPF_X:
+ case BPF_JMP | BPF_JSGT | BPF_X:
+ case BPF_JMP | BPF_JSGE | BPF_X:
+ emit(A64_CMP(1, dst, src), ctx);
+emit_cond_jmp:
+ jmp_offset = bpf2a64_offset(i + off, i, ctx);
+ check_imm19(jmp_offset);
+ switch (BPF_OP(code)) {
+ case BPF_JEQ:
+ jmp_cond = A64_COND_EQ;
+ break;
+ case BPF_JGT:
+ jmp_cond = A64_COND_HI;
+ break;
+ case BPF_JGE:
+ jmp_cond = A64_COND_CS;
+ break;
+ case BPF_JNE:
+ jmp_cond = A64_COND_NE;
+ break;
+ case BPF_JSGT:
+ jmp_cond = A64_COND_GT;
+ break;
+ case BPF_JSGE:
+ jmp_cond = A64_COND_GE;
+ break;
+ default:
+ return -EFAULT;
+ }
+ emit(A64_B_(jmp_cond, jmp_offset), ctx);
+ break;
+ case BPF_JMP | BPF_JSET | BPF_X:
+ emit(A64_TST(1, dst, src), ctx);
+ goto emit_cond_jmp;
+ /* IF (dst COND imm) JUMP off */
+ case BPF_JMP | BPF_JEQ | BPF_K:
+ case BPF_JMP | BPF_JGT | BPF_K:
+ case BPF_JMP | BPF_JGE | BPF_K:
+ case BPF_JMP | BPF_JNE | BPF_K:
+ case BPF_JMP | BPF_JSGT | BPF_K:
+ case BPF_JMP | BPF_JSGE | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(1, tmp, imm, ctx);
+ emit(A64_CMP(1, dst, tmp), ctx);
+ goto emit_cond_jmp;
+ case BPF_JMP | BPF_JSET | BPF_K:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(1, tmp, imm, ctx);
+ emit(A64_TST(1, dst, tmp), ctx);
+ goto emit_cond_jmp;
+ /* function call */
+ case BPF_JMP | BPF_CALL:
+ {
+ const u8 r0 = bpf2a64[BPF_REG_0];
+ const u64 func = (u64)__bpf_call_base + imm;
+
+ ctx->tmp_used = 1;
+ emit_a64_mov_i64(tmp, func, ctx);
+ emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+ emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+ emit(A64_BLR(tmp), ctx);
+ emit(A64_MOV(1, r0, A64_R(0)), ctx);
+ emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
+ break;
+ }
+ /* function return */
+ case BPF_JMP | BPF_EXIT:
+ if (i == ctx->prog->len - 1)
+ break;
+ jmp_offset = epilogue_offset(ctx);
+ check_imm26(jmp_offset);
+ emit(A64_B(jmp_offset), ctx);
+ break;
+
+ /* LDX: dst = *(size *)(src + off) */
+ case BPF_LDX | BPF_MEM | BPF_W:
+ case BPF_LDX | BPF_MEM | BPF_H:
+ case BPF_LDX | BPF_MEM | BPF_B:
+ case BPF_LDX | BPF_MEM | BPF_DW:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(1, tmp, off, ctx);
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ emit(A64_LDR32(dst, src, tmp), ctx);
+ break;
+ case BPF_H:
+ emit(A64_LDRH(dst, src, tmp), ctx);
+ break;
+ case BPF_B:
+ emit(A64_LDRB(dst, src, tmp), ctx);
+ break;
+ case BPF_DW:
+ emit(A64_LDR64(dst, src, tmp), ctx);
+ break;
+ }
+ break;
+
+ /* ST: *(size *)(dst + off) = imm */
+ case BPF_ST | BPF_MEM | BPF_W:
+ case BPF_ST | BPF_MEM | BPF_H:
+ case BPF_ST | BPF_MEM | BPF_B:
+ case BPF_ST | BPF_MEM | BPF_DW:
+ goto notyet;
+
+ /* STX: *(size *)(dst + off) = src */
+ case BPF_STX | BPF_MEM | BPF_W:
+ case BPF_STX | BPF_MEM | BPF_H:
+ case BPF_STX | BPF_MEM | BPF_B:
+ case BPF_STX | BPF_MEM | BPF_DW:
+ ctx->tmp_used = 1;
+ emit_a64_mov_i(1, tmp, off, ctx);
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ emit(A64_STR32(src, dst, tmp), ctx);
+ break;
+ case BPF_H:
+ emit(A64_STRH(src, dst, tmp), ctx);
+ break;
+ case BPF_B:
+ emit(A64_STRB(src, dst, tmp), ctx);
+ break;
+ case BPF_DW:
+ emit(A64_STR64(src, dst, tmp), ctx);
+ break;
+ }
+ break;
+ /* STX XADD: lock *(u32 *)(dst + off) += src */
+ case BPF_STX | BPF_XADD | BPF_W:
+ /* STX XADD: lock *(u64 *)(dst + off) += src */
+ case BPF_STX | BPF_XADD | BPF_DW:
+ goto notyet;
+
+ /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + imm)) */
+ case BPF_LD | BPF_ABS | BPF_W:
+ case BPF_LD | BPF_ABS | BPF_H:
+ case BPF_LD | BPF_ABS | BPF_B:
+ /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + src + imm)) */
+ case BPF_LD | BPF_IND | BPF_W:
+ case BPF_LD | BPF_IND | BPF_H:
+ case BPF_LD | BPF_IND | BPF_B:
+ {
+ const u8 r0 = bpf2a64[BPF_REG_0]; /* r0 = return value */
+ const u8 r6 = bpf2a64[BPF_REG_6]; /* r6 = pointer to sk_buff */
+ const u8 fp = bpf2a64[BPF_REG_FP];
+ const u8 r1 = bpf2a64[BPF_REG_1]; /* r1: struct sk_buff *skb */
+ const u8 r2 = bpf2a64[BPF_REG_2]; /* r2: int k */
+ const u8 r3 = bpf2a64[BPF_REG_3]; /* r3: unsigned int size */
+ const u8 r4 = bpf2a64[BPF_REG_4]; /* r4: void *buffer */
+ const u8 r5 = bpf2a64[BPF_REG_5]; /* r5: void *(*func)(...) */
+ int size;
+
+ emit(A64_MOV(1, r1, r6), ctx);
+ emit_a64_mov_i(0, r2, imm, ctx);
+ if (BPF_MODE(code) == BPF_IND)
+ emit(A64_ADD(0, r2, r2, src), ctx);
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ size = 4;
+ break;
+ case BPF_H:
+ size = 2;
+ break;
+ case BPF_B:
+ size = 1;
+ break;
+ default:
+ return -EINVAL;
+ }
+ emit_a64_mov_i64(r3, size, ctx);
+ emit(A64_ADD_I(1, r4, fp, MAX_BPF_STACK), ctx);
+ emit_a64_mov_i64(r5, (unsigned long)bpf_load_pointer, ctx);
+ emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx);
+ emit(A64_MOV(1, A64_FP, A64_SP), ctx);
+ emit(A64_BLR(r5), ctx);
+ emit(A64_MOV(1, r0, A64_R(0)), ctx);
+ emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
+
+ jmp_offset = epilogue_offset(ctx);
+ check_imm19(jmp_offset);
+ emit(A64_CBZ(1, r0, jmp_offset), ctx);
+ emit(A64_MOV(1, r5, r0), ctx);
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ emit(A64_LDR32(r0, r5, A64_ZR), ctx);
+#ifndef CONFIG_CPU_BIG_ENDIAN
+ emit(A64_REV32(0, r0, r0), ctx);
+#endif
+ break;
+ case BPF_H:
+ emit(A64_LDRH(r0, r5, A64_ZR), ctx);
+#ifndef CONFIG_CPU_BIG_ENDIAN
+ emit(A64_REV16(0, r0, r0), ctx);
+#endif
+ break;
+ case BPF_B:
+ emit(A64_LDRB(r0, r5, A64_ZR), ctx);
+ break;
+ }
+ break;
+ }
+notyet:
+ pr_info_once("*** NOT YET: opcode %02x ***\n", code);
+ return -EFAULT;
+
+ default:
+ pr_err_once("unknown opcode %02x\n", code);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int build_body(struct jit_ctx *ctx)
+{
+ const struct bpf_prog *prog = ctx->prog;
+ int i;
+
+ for (i = 0; i < prog->len; i++) {
+ const struct bpf_insn *insn = &prog->insnsi[i];
+ int ret;
+
+ if (ctx->image == NULL)
+ ctx->offset[i] = ctx->idx;
+
+ ret = build_insn(insn, ctx);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static inline void bpf_flush_icache(void *start, void *end)
+{
+ flush_icache_range((unsigned long)start, (unsigned long)end);
+}
+
+void bpf_jit_compile(struct bpf_prog *prog)
+{
+ /* Nothing to do here. We support Internal BPF. */
+}
+
+void bpf_int_jit_compile(struct bpf_prog *prog)
+{
+ struct jit_ctx ctx;
+ int image_size;
+
+ if (!bpf_jit_enable)
+ return;
+
+ if (!prog || !prog->len)
+ return;
+
+ memset(&ctx, 0, sizeof(ctx));
+ ctx.prog = prog;
+
+ ctx.offset = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
+ if (ctx.offset == NULL)
+ return;
+
+ /* 1. Initial fake pass to compute ctx->idx. */
+
+ /* Fake pass to fill in ctx->offset. */
+ if (build_body(&ctx))
+ goto out;
+
+ build_prologue(&ctx);
+
+ build_epilogue(&ctx);
+
+ /* Now we know the actual image size. */
+ image_size = sizeof(u32) * ctx.idx;
+ ctx.image = module_alloc(image_size);
+ if (unlikely(ctx.image == NULL))
+ goto out;
+
+ /* 2. Now, the actual pass. */
+
+ ctx.idx = 0;
+ build_prologue(&ctx);
+
+ ctx.body_offset = ctx.idx;
+ if (build_body(&ctx))
+ goto out;
+
+ build_epilogue(&ctx);
+
+ /* And we're done. */
+ if (bpf_jit_enable > 1)
+ bpf_jit_dump(prog->len, image_size, 2, ctx.image);
+
+ bpf_flush_icache(ctx.image, ctx.image + ctx.idx);
+ prog->bpf_func = (void *)ctx.image;
+ prog->jited = 1;
+
+out:
+ kfree(ctx.offset);
+}
+
+void bpf_jit_free(struct bpf_prog *prog)
+{
+ if (prog->jited)
+ module_free(NULL, prog->bpf_func);
+
+ kfree(prog);
+}
--
1.9.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox