* [PATCH 3.17-rc5] Fix confusing PFA_NO_NEW_PRIVS constant.
From: Tetsuo Handa @ 2014-09-20 10:40 UTC (permalink / raw)
To: keescook, linux-kernel
Cc: james.l.morris, oleg, luto, drysdale, mtk.manpages, wad, jln,
linux-api, x86, linux-arm-kernel, linux-mips, linux-arch,
linux-security-module
In-Reply-To: <541D16EA.70407@huawei.com>
Can you apply below patch before new PFA_* are defined?
Cgroups code might want to define PFA_SPREAD_PAGE as 1 and PFA_SPREAD_SLAB as 2.
----------------------------------------
>From 8543e68adb210142fa347d8bc9d83df0bb2c5291 Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Date: Sat, 20 Sep 2014 19:24:23 +0900
Subject: [PATCH 3.17-rc5] Fix confusing PFA_NO_NEW_PRIVS constant.
Commit 1d4457f99928 ("sched: move no_new_privs into new atomic flags")
defined PFA_NO_NEW_PRIVS as hexadecimal value, but it is confusing
because it is used as bit number. Redefine it as decimal bit number.
Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
---
include/linux/sched.h | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 5c2c885..4557765 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1957,7 +1957,7 @@ static inline void memalloc_noio_restore(unsigned int flags)
}
/* Per-process atomic flags. */
-#define PFA_NO_NEW_PRIVS 0x00000001 /* May not gain new privileges. */
+#define PFA_NO_NEW_PRIVS 0 /* May not gain new privileges. */
static inline bool task_no_new_privs(struct task_struct *p)
{
--
1.7.1
^ permalink raw reply related
* Re: [PATCH 3.17-rc5] Fix confusing PFA_NO_NEW_PRIVS constant.
From: Kees Cook @ 2014-09-20 17:19 UTC (permalink / raw)
To: Tetsuo Handa
Cc: LKML, James Morris, Oleg Nesterov, Andy Lutomirski,
David Drysdale, Michael Kerrisk-manpages, Will Drewry,
Julien Tinnes, Linux API,
x86-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
Linux MIPS Mailing List, linux-arch, linux-security-module
In-Reply-To: <201409201940.AHG21834.LJOFFHSFQOtVMO-JPay3/Yim36HaxMnTkn67Xf5DAMn2ifp@public.gmane.org>
On Sat, Sep 20, 2014 at 3:40 AM, Tetsuo Handa
<penguin-kernel-1yMVhJb1mP/7nzcFbJAaVXf5DAMn2ifp@public.gmane.org> wrote:
> Can you apply below patch before new PFA_* are defined?
> Cgroups code might want to define PFA_SPREAD_PAGE as 1 and PFA_SPREAD_SLAB as 2.
> ----------------------------------------
> >From 8543e68adb210142fa347d8bc9d83df0bb2c5291 Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa <penguin-kernel-JPay3/Yim36HaxMnTkn67Xf5DAMn2ifp@public.gmane.org>
> Date: Sat, 20 Sep 2014 19:24:23 +0900
> Subject: [PATCH 3.17-rc5] Fix confusing PFA_NO_NEW_PRIVS constant.
>
> Commit 1d4457f99928 ("sched: move no_new_privs into new atomic flags")
> defined PFA_NO_NEW_PRIVS as hexadecimal value, but it is confusing
> because it is used as bit number. Redefine it as decimal bit number.
>
> Signed-off-by: Tetsuo Handa <penguin-kernel-JPay3/Yim36HaxMnTkn67Xf5DAMn2ifp@public.gmane.org>
> ---
> include/linux/sched.h | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 5c2c885..4557765 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -1957,7 +1957,7 @@ static inline void memalloc_noio_restore(unsigned int flags)
> }
>
> /* Per-process atomic flags. */
> -#define PFA_NO_NEW_PRIVS 0x00000001 /* May not gain new privileges. */
> +#define PFA_NO_NEW_PRIVS 0 /* May not gain new privileges. */
>
> static inline bool task_no_new_privs(struct task_struct *p)
> {
Thanks, good catch.
Acked-by: Kees Cook <keescook-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org>
-Kees
--
Kees Cook
Chrome OS Security
^ permalink raw reply
* [PATCH v14 net-next 00/11] eBPF syscall, verifier, testsuite
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api-u79uwXL29TY76Z2rM5mHXA,
netdev-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA
Hi All,
v13 -> v14:
- small change to 1st patch to ease 'new userspace with old kernel'
problem (done similar to perf_copy_attr()) (suggested by Daniel)
- the rest unchanged
v12 -> v13:
- replaced 'foo __user *' pointers with __aligned_u64 (suggested by David)
- added __attribute__((aligned(8)) to 'union bpf_attr' to keep
constant alignment between patches
- updated manpage and syscall wrappers due to __aligned_u64
- rebased, retested on x64 with 32-bit and 64-bit userspace and on i386,
build tested on arm32,sparc64
v11 -> v12:
- dropped patch 11 and copied few macros to libbpf.h (suggested by Daniel)
- replaced 'enum bpf_prog_type' with u32 to be safe in compat (.. Andy)
- implemented and tested compat support (not part of this set) (.. Daniel)
- changed 'void *log_buf' to 'char *' (.. Daniel)
- combined struct bpf_work_struct and bpf_prog_info (.. Daniel)
- added better return value explanation to manpage (.. Andy)
- added log_buf/log_size explanation to manpage (.. Andy & Daniel)
- added a lot more info about prog_type and map_type to manpage (.. Andy)
- rebased, tweaked test_stubs
Patches 1-4 establish BPF syscall shell for maps and programs.
Patches 5-10 add verifier step by step
Patch 11 adds test stubs for 'unspec' program type and verifier testsuite
from user space
Note that patches 1,3,4,7 add commands and attributes to the syscall
while being backwards compatible from each other, which should demonstrate
how other commands can be added in the future.
After this set the programs can be loaded for testing only. They cannot
be attached to any events. Though manpage talks about tracing and sockets,
it will be a subject of future patches.
Please take a look at manpage:
BPF(2) Linux Programmer's Manual BPF(2)
NAME
bpf - perform a command on eBPF map or program
SYNOPSIS
#include <linux/bpf.h>
int bpf(int cmd, union bpf_attr *attr, unsigned int size);
DESCRIPTION
bpf() syscall is a multiplexor for a range of different operations on
eBPF which can be characterized as "universal in-kernel virtual
machine". eBPF is similar to original Berkeley Packet Filter (or
"classic BPF") used to filter network packets. Both statically analyze
the programs before loading them into the kernel to ensure that
programs cannot harm the running system.
eBPF extends classic BPF in multiple ways including ability to call in-
kernel helper functions and access shared data structures like eBPF
maps. The programs can be written in a restricted C that is compiled
into eBPF bytecode and executed on the eBPF virtual machine or JITed
into native instruction set.
eBPF Design/Architecture
eBPF maps is a generic storage of different types. User process can
create multiple maps (with key/value being opaque bytes of data) and
access them via file descriptor. In parallel eBPF programs can access
maps from inside the kernel. It's up to user process and eBPF program
to decide what they store inside maps.
eBPF programs are similar to kernel modules. They are loaded by the
user process and automatically unloaded when process exits. Each eBPF
program is a safe run-to-completion set of instructions. eBPF verifier
statically determines that the program terminates and is safe to
execute. During verification the program takes a hold of maps that it
intends to use, so selected maps cannot be removed until the program is
unloaded. The program can be attached to different events. These events
can be packets, tracepoint events and other types in the future. A new
event triggers execution of the program which may store information
about the event in the maps. Beyond storing data the programs may call
into in-kernel helper functions which may, for example, dump stack, do
trace_printk or other forms of live kernel debugging. The same program
can be attached to multiple events. Different programs can access the
same map:
tracepoint tracepoint tracepoint sk_buff sk_buff
event A event B event C on eth0 on eth1
| | | | |
| | | | |
--> tracing <-- tracing socket socket
prog_1 prog_2 prog_3 prog_4
| | | |
|--- -----| |-------| map_3
map_1 map_2
Syscall Arguments
bpf() syscall operation is determined by cmd which can be one of the
following:
BPF_MAP_CREATE
Create a map with given type and attributes and return map FD
BPF_MAP_LOOKUP_ELEM
Lookup element by key in a given map and return its value
BPF_MAP_UPDATE_ELEM
Create or update element (key/value pair) in a given map
BPF_MAP_DELETE_ELEM
Lookup and delete element by key in a given map
BPF_MAP_GET_NEXT_KEY
Lookup element by key in a given map and return key of next
element
BPF_PROG_LOAD
Verify and load eBPF program
attr is a pointer to a union of type bpf_attr as defined below.
size is the size of the union.
union bpf_attr {
struct { /* anonymous struct used by BPF_MAP_CREATE command */
__u32 map_type;
__u32 key_size; /* size of key in bytes */
__u32 value_size; /* size of value in bytes */
__u32 max_entries; /* max number of entries in a map */
};
struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */
__u32 map_fd;
__aligned_u64 key;
union {
__aligned_u64 value;
__aligned_u64 next_key;
};
};
struct { /* anonymous struct used by BPF_PROG_LOAD command */
__u32 prog_type;
__u32 insn_cnt;
__aligned_u64 insns; /* 'const struct bpf_insn *' */
__aligned_u64 license; /* 'const char *' */
__u32 log_level; /* verbosity level of eBPF verifier */
__u32 log_size; /* size of user buffer */
__aligned_u64 log_buf; /* user supplied 'char *' buffer */
};
} __attribute__((aligned(8)));
eBPF maps
maps is a generic storage of different types for sharing data between
kernel and userspace.
Any map type has the following attributes:
. type
. max number of elements
. key size in bytes
. value size in bytes
The following wrapper functions demonstrate how this syscall can be
used to access the maps. The functions use the cmd argument to invoke
different operations.
BPF_MAP_CREATE
int bpf_create_map(enum bpf_map_type map_type, int key_size,
int value_size, int max_entries)
{
union bpf_attr attr = {
.map_type = map_type,
.key_size = key_size,
.value_size = value_size,
.max_entries = max_entries
};
return bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
}
bpf() syscall creates a map of map_type type and given
attributes key_size, value_size, max_entries. On success it
returns process-local file descriptor. On error, -1 is returned
and errno is set to EINVAL or EPERM or ENOMEM.
The attributes key_size and value_size will be used by verifier
during program loading to check that program is calling
bpf_map_*_elem() helper functions with correctly initialized key
and that program doesn't access map element value beyond
specified value_size. For example, when map is created with
key_size = 8 and program does:
bpf_map_lookup_elem(map_fd, fp - 4)
such program will be rejected, since in-kernel helper function
bpf_map_lookup_elem(map_fd, void *key) expects to read 8 bytes
from 'key' pointer, but 'fp - 4' starting address will cause out
of bounds stack access.
Similarly, when map is created with value_size = 1 and program
does:
value = bpf_map_lookup_elem(...);
*(u32 *)value = 1;
such program will be rejected, since it accesses value pointer
beyond specified 1 byte value_size limit.
Currently only hash table map_type is supported:
enum bpf_map_type {
BPF_MAP_TYPE_UNSPEC,
BPF_MAP_TYPE_HASH,
};
map_type selects one of the available map implementations in
kernel. For all map_types eBPF programs access maps with the
same bpf_map_lookup_elem()/bpf_map_update_elem() helper
functions.
BPF_MAP_LOOKUP_ELEM
int bpf_lookup_elem(int fd, void *key, void *value)
{
union bpf_attr attr = {
.map_fd = fd,
.key = ptr_to_u64(key),
.value = ptr_to_u64(value),
};
return bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
}
bpf() syscall looks up an element with given key in a map fd.
If element is found it returns zero and stores element's value
into value. If element is not found it returns -1 and sets
errno to ENOENT.
BPF_MAP_UPDATE_ELEM
int bpf_update_elem(int fd, void *key, void *value)
{
union bpf_attr attr = {
.map_fd = fd,
.key = ptr_to_u64(key),
.value = ptr_to_u64(value),
};
return bpf(BPF_MAP_UPDATE_ELEM, &attr, sizeof(attr));
}
The call creates or updates element with given key/value in a
map fd. On success it returns zero. On error, -1 is returned
and errno is set to EINVAL or EPERM or ENOMEM or E2BIG. E2BIG
indicates that number of elements in the map reached max_entries
limit specified at map creation time.
BPF_MAP_DELETE_ELEM
int bpf_delete_elem(int fd, void *key)
{
union bpf_attr attr = {
.map_fd = fd,
.key = ptr_to_u64(key),
};
return bpf(BPF_MAP_DELETE_ELEM, &attr, sizeof(attr));
}
The call deletes an element in a map fd with given key. Returns
zero on success. If element is not found it returns -1 and sets
errno to ENOENT.
BPF_MAP_GET_NEXT_KEY
int bpf_get_next_key(int fd, void *key, void *next_key)
{
union bpf_attr attr = {
.map_fd = fd,
.key = ptr_to_u64(key),
.next_key = ptr_to_u64(next_key),
};
return bpf(BPF_MAP_GET_NEXT_KEY, &attr, sizeof(attr));
}
The call looks up an element by key in a given map fd and
returns key of the next element into next_key pointer. If key is
not found, it return zero and returns key of the first element
into next_key. If key is the last element, it returns -1 and
sets errno to ENOENT. Other possible errno values are ENOMEM,
EFAULT, EPERM, EINVAL. This method can be used to iterate over
all elements of the map.
close(map_fd)
will delete the map map_fd. Exiting process will delete all
maps automatically.
eBPF programs
BPF_PROG_LOAD
This cmd is used to load eBPF program into the kernel.
char bpf_log_buf[LOG_BUF_SIZE];
int bpf_prog_load(enum bpf_prog_type prog_type,
const struct bpf_insn *insns, int insn_cnt,
const char *license)
{
union bpf_attr attr = {
.prog_type = prog_type,
.insns = ptr_to_u64(insns),
.insn_cnt = insn_cnt,
.license = ptr_to_u64(license),
.log_buf = ptr_to_u64(bpf_log_buf),
.log_size = LOG_BUF_SIZE,
.log_level = 1,
};
return bpf(BPF_PROG_LOAD, &attr, sizeof(attr));
}
prog_type is one of the available program types:
enum bpf_prog_type {
BPF_PROG_TYPE_UNSPEC,
BPF_PROG_TYPE_SOCKET,
BPF_PROG_TYPE_TRACING,
};
By picking prog_type program author selects a set of helper
functions callable from eBPF program and corresponding format of
struct bpf_context (which is the data blob passed into the
program as the first argument). For example, the programs
loaded with prog_type = TYPE_TRACING may call bpf_printk()
helper, whereas TYPE_SOCKET programs may not. The set of
functions available to the programs under given type may
increase in the future.
Currently the set of functions for TYPE_TRACING is:
bpf_map_lookup_elem(map_fd, void *key) // lookup key in a map_fd
bpf_map_update_elem(map_fd, void *key, void *value) // update key/value
bpf_map_delete_elem(map_fd, void *key) // delete key in a map_fd
bpf_ktime_get_ns(void) // returns current ktime
bpf_printk(char *fmt, int fmt_size, ...) // prints into trace buffer
bpf_memcmp(void *ptr1, void *ptr2, int size) // non-faulting memcmp
bpf_fetch_ptr(void *ptr) // non-faulting load pointer from any address
bpf_fetch_u8(void *ptr) // non-faulting 1 byte load
bpf_fetch_u16(void *ptr) // other non-faulting loads
bpf_fetch_u32(void *ptr)
bpf_fetch_u64(void *ptr)
and bpf_context is defined as:
struct bpf_context {
/* argN fields match one to one to arguments passed to trace events */
u64 arg1, arg2, arg3, arg4, arg5, arg6;
/* return value from kretprobe event or from syscall_exit event */
u64 ret;
};
The set of helper functions for TYPE_SOCKET is TBD.
More program types may be added in the future. Like
BPF_PROG_TYPE_USER_TRACING for unprivileged programs.
BPF_PROG_TYPE_UNSPEC is used for testing only. Such programs
cannot be attached to events.
insns array of "struct bpf_insn" instructions
insn_cnt number of instructions in the program
license license string, which must be GPL compatible to call
helper functions marked gpl_only
log_buf user supplied buffer that in-kernel verifier is using to
store verification log. Log is a multi-line string that should
be used by program author to understand how verifier came to
conclusion that program is unsafe. The format of the output can
change at any time as verifier evolves.
log_size size of user buffer. If size of the buffer is not large
enough to store all verifier messages, -1 is returned and errno
is set to ENOSPC.
log_level verbosity level of eBPF verifier, where zero means no
logs provided
close(prog_fd)
will unload eBPF program
The maps are accesible from programs and generally tie the two
together. Programs process various events (like tracepoint, kprobe,
packets) and store the data into maps. User space fetches data from
maps. Either the same or a different map may be used by user space as
configuration space to alter program behavior on the fly.
Events
Once an eBPF program is loaded, it can be attached to an event. Various
kernel subsystems have different ways to do so. For example:
setsockopt(sock, SOL_SOCKET, SO_ATTACH_BPF, &prog_fd, sizeof(prog_fd));
will attach the program prog_fd to socket sock which was received by
prior call to socket().
ioctl(event_fd, PERF_EVENT_IOC_SET_BPF, prog_fd);
will attach the program prog_fd to perf event event_fd which was
received by prior call to perf_event_open().
Another way to attach the program to a tracing event is:
event_fd = open("/sys/kernel/debug/tracing/events/skb/kfree_skb/filter");
write(event_fd, "bpf-123"); /* where 123 is eBPF program FD */
/* here program is attached and will be triggered by events */
close(event_fd); /* to detach from event */
EXAMPLES
/* eBPF+sockets example:
* 1. create map with maximum of 2 elements
* 2. set map[6] = 0 and map[17] = 0
* 3. load eBPF program that counts number of TCP and UDP packets received
* via map[skb->ip->proto]++
* 4. attach prog_fd to raw socket via setsockopt()
* 5. print number of received TCP/UDP packets every second
*/
int main(int ac, char **av)
{
int sock, map_fd, prog_fd, key;
long long value = 0, tcp_cnt, udp_cnt;
map_fd = bpf_create_map(BPF_MAP_TYPE_HASH, sizeof(key), sizeof(value), 2);
if (map_fd < 0) {
printf("failed to create map '%s'\n", strerror(errno));
/* likely not run as root */
return 1;
}
key = 6; /* ip->proto == tcp */
assert(bpf_update_elem(map_fd, &key, &value) == 0);
key = 17; /* ip->proto == udp */
assert(bpf_update_elem(map_fd, &key, &value) == 0);
struct bpf_insn prog[] = {
BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), /* r6 = r1 */
BPF_LD_ABS(BPF_B, 14 + 9), /* r0 = ip->proto */
BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_0, -4),/* *(u32 *)(fp - 4) = r0 */
BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), /* r2 = fp */
BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), /* r2 = r2 - 4 */
BPF_LD_MAP_FD(BPF_REG_1, map_fd), /* r1 = map_fd */
BPF_CALL_FUNC(BPF_FUNC_map_lookup_elem), /* r0 = map_lookup(r1, r2) */
BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), /* if (r0 == 0) goto pc+2 */
BPF_MOV64_IMM(BPF_REG_1, 1), /* r1 = 1 */
BPF_XADD(BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0), /* lock *(u64 *)r0 += r1 */
BPF_MOV64_IMM(BPF_REG_0, 0), /* r0 = 0 */
BPF_EXIT_INSN(), /* return r0 */
};
prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET, prog, sizeof(prog), "GPL");
assert(prog_fd >= 0);
sock = open_raw_sock("lo");
assert(setsockopt(sock, SOL_SOCKET, SO_ATTACH_BPF, &prog_fd,
sizeof(prog_fd)) == 0);
for (;;) {
key = 6;
assert(bpf_lookup_elem(map_fd, &key, &tcp_cnt) == 0);
key = 17;
assert(bpf_lookup_elem(map_fd, &key, &udp_cnt) == 0);
printf("TCP %lld UDP %lld packets0, tcp_cnt, udp_cnt);
sleep(1);
}
return 0;
}
RETURN VALUE
For a successful call, the return value depends on the operation:
BPF_MAP_CREATE
The new file descriptor associated with eBPF map.
BPF_PROG_LOAD
The new file descriptor associated with eBPF program.
All other commands
Zero.
On error, -1 is returned, and errno is set appropriately.
ERRORS
EPERM bpf() syscall was made without sufficient privilege (without the
CAP_SYS_ADMIN capability).
ENOMEM Cannot allocate sufficient memory.
EBADF fd is not an open file descriptor
EFAULT One of the pointers ( key or value or log_buf or insns ) is
outside accessible address space.
EINVAL The value specified in cmd is not recognized by this kernel.
EINVAL For BPF_MAP_CREATE, either map_type or attributes are invalid.
EINVAL For BPF_MAP_*_ELEM commands, some of the fields of "union
bpf_attr" unused by this command are not set to zero.
EINVAL For BPF_PROG_LOAD, attempt to load invalid program (unrecognized
instruction or uses reserved fields or jumps out of range or
loop detected or calls unknown function).
EACCES For BPF_PROG_LOAD, though program has valid instructions, it was
rejected, since it was deemed unsafe (may access disallowed
memory region or uninitialized stack/register or function
constraints don't match actual types or misaligned access). In
such case it is recommended to call bpf() again with log_level =
1 and examine log_buf for specific reason provided by verifier.
ENOENT For BPF_MAP_LOOKUP_ELEM or BPF_MAP_DELETE_ELEM, indicates that
element with given key was not found.
E2BIG program is too large or a map reached max_entries limit (max
number of elements).
NOTES
These commands may be used only by a privileged process (one having the
CAP_SYS_ADMIN capability).
SEE ALSO
eBPF architecture and instruction set is explained in
Documentation/networking/filter.txt
Linux 2014-09-16 BPF(2)
^ permalink raw reply
* [PATCH v14 net-next 01/11] bpf: introduce BPF syscall and maps
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
BPF syscall is a multiplexor for a range of different operations on eBPF.
This patch introduces syscall with single command to create a map.
Next patch adds commands to access maps.
'maps' is a generic storage of different types for sharing data between kernel
and userspace.
Userspace example:
/* this syscall wrapper creates a map with given type and attributes
* and returns map_fd on success.
* use close(map_fd) to delete the map
*/
int bpf_create_map(enum bpf_map_type map_type, int key_size,
int value_size, int max_entries)
{
union bpf_attr attr = {
.map_type = map_type,
.key_size = key_size,
.value_size = value_size,
.max_entries = max_entries
};
return bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
}
'union bpf_attr' is backwards compatible with future extensions.
More details in Documentation/networking/filter.txt and in manpage
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
Documentation/networking/filter.txt | 39 ++++++++
include/linux/bpf.h | 41 +++++++++
include/uapi/linux/bpf.h | 23 +++++
kernel/bpf/Makefile | 2 +-
kernel/bpf/syscall.c | 169 +++++++++++++++++++++++++++++++++++
5 files changed, 273 insertions(+), 1 deletion(-)
create mode 100644 include/linux/bpf.h
create mode 100644 kernel/bpf/syscall.c
diff --git a/Documentation/networking/filter.txt b/Documentation/networking/filter.txt
index 81916ab5d96f..1900d29518f1 100644
--- a/Documentation/networking/filter.txt
+++ b/Documentation/networking/filter.txt
@@ -1001,6 +1001,45 @@ instruction that loads 64-bit immediate value into a dst_reg.
Classic BPF has similar instruction: BPF_LD | BPF_W | BPF_IMM which loads
32-bit immediate value into a register.
+eBPF maps
+---------
+'maps' is a generic storage of different types for sharing data between kernel
+and userspace.
+
+The maps are accessed from user space via BPF syscall, which has commands:
+- create a map with given type and attributes
+ map_fd = bpf(BPF_MAP_CREATE, union bpf_attr *attr, u32 size)
+ using attr->map_type, attr->key_size, attr->value_size, attr->max_entries
+ returns process-local file descriptor or negative error
+
+- lookup key in a given map
+ err = bpf(BPF_MAP_LOOKUP_ELEM, union bpf_attr *attr, u32 size)
+ using attr->map_fd, attr->key, attr->value
+ returns zero and stores found elem into value or negative error
+
+- create or update key/value pair in a given map
+ err = bpf(BPF_MAP_UPDATE_ELEM, union bpf_attr *attr, u32 size)
+ using attr->map_fd, attr->key, attr->value
+ returns zero or negative error
+
+- find and delete element by key in a given map
+ err = bpf(BPF_MAP_DELETE_ELEM, union bpf_attr *attr, u32 size)
+ using attr->map_fd, attr->key
+
+- to delete map: close(fd)
+ Exiting process will delete maps automatically
+
+userspace programs use this syscall to create/access maps that eBPF programs
+are concurrently updating.
+
+maps can have different types: hash, array, bloom filter, radix-tree, etc.
+
+The map is defined by:
+ . type
+ . max number of elements
+ . key size in bytes
+ . value size in bytes
+
Testing
-------
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
new file mode 100644
index 000000000000..48014a71f0fe
--- /dev/null
+++ b/include/linux/bpf.h
@@ -0,0 +1,41 @@
+/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#ifndef _LINUX_BPF_H
+#define _LINUX_BPF_H 1
+
+#include <uapi/linux/bpf.h>
+#include <linux/workqueue.h>
+
+struct bpf_map;
+
+/* map is generic key/value storage optionally accesible by eBPF programs */
+struct bpf_map_ops {
+ /* funcs callable from userspace (via syscall) */
+ struct bpf_map *(*map_alloc)(union bpf_attr *attr);
+ void (*map_free)(struct bpf_map *);
+};
+
+struct bpf_map {
+ atomic_t refcnt;
+ enum bpf_map_type map_type;
+ u32 key_size;
+ u32 value_size;
+ u32 max_entries;
+ struct bpf_map_ops *ops;
+ struct work_struct work;
+};
+
+struct bpf_map_type_list {
+ struct list_head list_node;
+ struct bpf_map_ops *ops;
+ enum bpf_map_type type;
+};
+
+void bpf_register_map_type(struct bpf_map_type_list *tl);
+void bpf_map_put(struct bpf_map *map);
+
+#endif /* _LINUX_BPF_H */
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 479ed0b6be16..f58a10f9670c 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -62,4 +62,27 @@ struct bpf_insn {
__s32 imm; /* signed immediate constant */
};
+/* BPF syscall commands */
+enum bpf_cmd {
+ /* create a map with given type and attributes
+ * fd = bpf(BPF_MAP_CREATE, union bpf_attr *, u32 size)
+ * returns fd or negative error
+ * map is deleted when fd is closed
+ */
+ BPF_MAP_CREATE,
+};
+
+enum bpf_map_type {
+ BPF_MAP_TYPE_UNSPEC,
+};
+
+union bpf_attr {
+ struct { /* anonymous struct used by BPF_MAP_CREATE command */
+ __u32 map_type; /* one of enum bpf_map_type */
+ __u32 key_size; /* size of key in bytes */
+ __u32 value_size; /* size of value in bytes */
+ __u32 max_entries; /* max number of entries in a map */
+ };
+} __attribute__((aligned(8)));
+
#endif /* _UAPI__LINUX_BPF_H__ */
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 6a71145e2769..e9f7334ed07a 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -1 +1 @@
-obj-y := core.o
+obj-y := core.o syscall.o
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
new file mode 100644
index 000000000000..428a0e23adc0
--- /dev/null
+++ b/kernel/bpf/syscall.c
@@ -0,0 +1,169 @@
+/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include <linux/bpf.h>
+#include <linux/syscalls.h>
+#include <linux/slab.h>
+#include <linux/anon_inodes.h>
+
+static LIST_HEAD(bpf_map_types);
+
+static struct bpf_map *find_and_alloc_map(union bpf_attr *attr)
+{
+ struct bpf_map_type_list *tl;
+ struct bpf_map *map;
+
+ list_for_each_entry(tl, &bpf_map_types, list_node) {
+ if (tl->type == attr->map_type) {
+ map = tl->ops->map_alloc(attr);
+ if (IS_ERR(map))
+ return map;
+ map->ops = tl->ops;
+ map->map_type = attr->map_type;
+ return map;
+ }
+ }
+ return ERR_PTR(-EINVAL);
+}
+
+/* boot time registration of different map implementations */
+void bpf_register_map_type(struct bpf_map_type_list *tl)
+{
+ list_add(&tl->list_node, &bpf_map_types);
+}
+
+/* called from workqueue */
+static void bpf_map_free_deferred(struct work_struct *work)
+{
+ struct bpf_map *map = container_of(work, struct bpf_map, work);
+
+ /* implementation dependent freeing */
+ map->ops->map_free(map);
+}
+
+/* decrement map refcnt and schedule it for freeing via workqueue
+ * (unrelying map implementation ops->map_free() might sleep)
+ */
+void bpf_map_put(struct bpf_map *map)
+{
+ if (atomic_dec_and_test(&map->refcnt)) {
+ INIT_WORK(&map->work, bpf_map_free_deferred);
+ schedule_work(&map->work);
+ }
+}
+
+static int bpf_map_release(struct inode *inode, struct file *filp)
+{
+ struct bpf_map *map = filp->private_data;
+
+ bpf_map_put(map);
+ return 0;
+}
+
+static const struct file_operations bpf_map_fops = {
+ .release = bpf_map_release,
+};
+
+/* helper macro to check that unused fields 'union bpf_attr' are zero */
+#define CHECK_ATTR(CMD) \
+ memchr_inv((void *) &attr->CMD##_LAST_FIELD + \
+ sizeof(attr->CMD##_LAST_FIELD), 0, \
+ sizeof(*attr) - \
+ offsetof(union bpf_attr, CMD##_LAST_FIELD) - \
+ sizeof(attr->CMD##_LAST_FIELD)) != NULL
+
+#define BPF_MAP_CREATE_LAST_FIELD max_entries
+/* called via syscall */
+static int map_create(union bpf_attr *attr)
+{
+ struct bpf_map *map;
+ int err;
+
+ err = CHECK_ATTR(BPF_MAP_CREATE);
+ if (err)
+ return -EINVAL;
+
+ /* find map type and init map: hashtable vs rbtree vs bloom vs ... */
+ map = find_and_alloc_map(attr);
+ if (IS_ERR(map))
+ return PTR_ERR(map);
+
+ atomic_set(&map->refcnt, 1);
+
+ err = anon_inode_getfd("bpf-map", &bpf_map_fops, map, O_RDWR | O_CLOEXEC);
+
+ if (err < 0)
+ /* failed to allocate fd */
+ goto free_map;
+
+ return err;
+
+free_map:
+ map->ops->map_free(map);
+ return err;
+}
+
+SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
+{
+ union bpf_attr attr = {};
+ int err;
+
+ /* the syscall is limited to root temporarily. This restriction will be
+ * lifted when security audit is clean. Note that eBPF+tracing must have
+ * this restriction, since it may pass kernel data to user space
+ */
+ if (!capable(CAP_SYS_ADMIN))
+ return -EPERM;
+
+ if (!access_ok(VERIFY_READ, uattr, 1))
+ return -EFAULT;
+
+ if (size > PAGE_SIZE) /* silly large */
+ return -E2BIG;
+
+ /* If we're handed a bigger struct than we know of,
+ * ensure all the unknown bits are 0 - i.e. new
+ * user-space does not rely on any kernel feature
+ * extensions we dont know about yet.
+ */
+ if (size > sizeof(attr)) {
+ unsigned char __user *addr;
+ unsigned char __user *end;
+ unsigned char val;
+
+ addr = (void __user *)uattr + sizeof(attr);
+ end = (void __user *)uattr + size;
+
+ for (; addr < end; addr++) {
+ err = get_user(val, addr);
+ if (err)
+ return err;
+ if (val)
+ return -E2BIG;
+ }
+ size = sizeof(attr);
+ }
+
+ /* copy attributes from user space, may be less than sizeof(bpf_attr) */
+ if (copy_from_user(&attr, uattr, size) != 0)
+ return -EFAULT;
+
+ switch (cmd) {
+ case BPF_MAP_CREATE:
+ err = map_create(&attr);
+ break;
+ default:
+ err = -EINVAL;
+ break;
+ }
+
+ return err;
+}
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 02/11] bpf: enable bpf syscall on x64 and i386
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
done as separate commit to ease conflict resolution
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
arch/x86/syscalls/syscall_32.tbl | 1 +
arch/x86/syscalls/syscall_64.tbl | 1 +
include/linux/syscalls.h | 3 ++-
include/uapi/asm-generic/unistd.h | 4 +++-
kernel/sys_ni.c | 3 +++
5 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/arch/x86/syscalls/syscall_32.tbl b/arch/x86/syscalls/syscall_32.tbl
index 028b78168d85..9fe1b5d002f0 100644
--- a/arch/x86/syscalls/syscall_32.tbl
+++ b/arch/x86/syscalls/syscall_32.tbl
@@ -363,3 +363,4 @@
354 i386 seccomp sys_seccomp
355 i386 getrandom sys_getrandom
356 i386 memfd_create sys_memfd_create
+357 i386 bpf sys_bpf
diff --git a/arch/x86/syscalls/syscall_64.tbl b/arch/x86/syscalls/syscall_64.tbl
index 35dd922727b9..281150b539a2 100644
--- a/arch/x86/syscalls/syscall_64.tbl
+++ b/arch/x86/syscalls/syscall_64.tbl
@@ -327,6 +327,7 @@
318 common getrandom sys_getrandom
319 common memfd_create sys_memfd_create
320 common kexec_file_load sys_kexec_file_load
+321 common bpf sys_bpf
#
# x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 0f86d85a9ce4..bda9b81357cc 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -65,6 +65,7 @@ struct old_linux_dirent;
struct perf_event_attr;
struct file_handle;
struct sigaltstack;
+union bpf_attr;
#include <linux/types.h>
#include <linux/aio_abi.h>
@@ -875,5 +876,5 @@ asmlinkage long sys_seccomp(unsigned int op, unsigned int flags,
const char __user *uargs);
asmlinkage long sys_getrandom(char __user *buf, size_t count,
unsigned int flags);
-
+asmlinkage long sys_bpf(int cmd, union bpf_attr *attr, unsigned int size);
#endif
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 11d11bc5c78f..22749c134117 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -705,9 +705,11 @@ __SYSCALL(__NR_seccomp, sys_seccomp)
__SYSCALL(__NR_getrandom, sys_getrandom)
#define __NR_memfd_create 279
__SYSCALL(__NR_memfd_create, sys_memfd_create)
+#define __NR_bpf 280
+__SYSCALL(__NR_bpf, sys_bpf)
#undef __NR_syscalls
-#define __NR_syscalls 280
+#define __NR_syscalls 281
/*
* All syscalls below here should go away really,
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index 391d4ddb6f4b..b4b5083f5f5e 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -218,3 +218,6 @@ cond_syscall(sys_kcmp);
/* operate on Secure Computing state */
cond_syscall(sys_seccomp);
+
+/* access BPF programs and maps */
+cond_syscall(sys_bpf);
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 03/11] bpf: add lookup/update/delete/iterate methods to BPF maps
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
'maps' is a generic storage of different types for sharing data between kernel
and userspace.
The maps are accessed from user space via BPF syscall, which has commands:
- create a map with given type and attributes
fd = bpf(BPF_MAP_CREATE, union bpf_attr *attr, u32 size)
returns fd or negative error
- lookup key in a given map referenced by fd
err = bpf(BPF_MAP_LOOKUP_ELEM, union bpf_attr *attr, u32 size)
using attr->map_fd, attr->key, attr->value
returns zero and stores found elem into value or negative error
- create or update key/value pair in a given map
err = bpf(BPF_MAP_UPDATE_ELEM, union bpf_attr *attr, u32 size)
using attr->map_fd, attr->key, attr->value
returns zero or negative error
- find and delete element by key in a given map
err = bpf(BPF_MAP_DELETE_ELEM, union bpf_attr *attr, u32 size)
using attr->map_fd, attr->key
- iterate map elements (based on input key return next_key)
err = bpf(BPF_MAP_GET_NEXT_KEY, union bpf_attr *attr, u32 size)
using attr->map_fd, attr->key, attr->next_key
- close(fd) deletes the map
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
include/linux/bpf.h | 8 ++
include/uapi/linux/bpf.h | 38 ++++++++
kernel/bpf/syscall.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 281 insertions(+)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 48014a71f0fe..2887f3f9da59 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -9,6 +9,7 @@
#include <uapi/linux/bpf.h>
#include <linux/workqueue.h>
+#include <linux/file.h>
struct bpf_map;
@@ -17,6 +18,12 @@ struct bpf_map_ops {
/* funcs callable from userspace (via syscall) */
struct bpf_map *(*map_alloc)(union bpf_attr *attr);
void (*map_free)(struct bpf_map *);
+ int (*map_get_next_key)(struct bpf_map *map, void *key, void *next_key);
+
+ /* funcs callable from userspace and from eBPF programs */
+ void *(*map_lookup_elem)(struct bpf_map *map, void *key);
+ int (*map_update_elem)(struct bpf_map *map, void *key, void *value);
+ int (*map_delete_elem)(struct bpf_map *map, void *key);
};
struct bpf_map {
@@ -37,5 +44,6 @@ struct bpf_map_type_list {
void bpf_register_map_type(struct bpf_map_type_list *tl);
void bpf_map_put(struct bpf_map *map);
+struct bpf_map *bpf_map_get(struct fd f);
#endif /* _LINUX_BPF_H */
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index f58a10f9670c..395cabd2ca0a 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -70,6 +70,35 @@ enum bpf_cmd {
* map is deleted when fd is closed
*/
BPF_MAP_CREATE,
+
+ /* lookup key in a given map
+ * err = bpf(BPF_MAP_LOOKUP_ELEM, union bpf_attr *attr, u32 size)
+ * Using attr->map_fd, attr->key, attr->value
+ * returns zero and stores found elem into value
+ * or negative error
+ */
+ BPF_MAP_LOOKUP_ELEM,
+
+ /* create or update key/value pair in a given map
+ * err = bpf(BPF_MAP_UPDATE_ELEM, union bpf_attr *attr, u32 size)
+ * Using attr->map_fd, attr->key, attr->value
+ * returns zero or negative error
+ */
+ BPF_MAP_UPDATE_ELEM,
+
+ /* find and delete elem by key in a given map
+ * err = bpf(BPF_MAP_DELETE_ELEM, union bpf_attr *attr, u32 size)
+ * Using attr->map_fd, attr->key
+ * returns zero or negative error
+ */
+ BPF_MAP_DELETE_ELEM,
+
+ /* lookup key in a given map and return next key
+ * err = bpf(BPF_MAP_GET_NEXT_KEY, union bpf_attr *attr, u32 size)
+ * Using attr->map_fd, attr->key, attr->next_key
+ * returns zero and stores next key or negative error
+ */
+ BPF_MAP_GET_NEXT_KEY,
};
enum bpf_map_type {
@@ -83,6 +112,15 @@ union bpf_attr {
__u32 value_size; /* size of value in bytes */
__u32 max_entries; /* max number of entries in a map */
};
+
+ struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */
+ __u32 map_fd;
+ __aligned_u64 key;
+ union {
+ __aligned_u64 value;
+ __aligned_u64 next_key;
+ };
+ };
} __attribute__((aligned(8)));
#endif /* _UAPI__LINUX_BPF_H__ */
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 428a0e23adc0..f94349ecaf61 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -13,6 +13,7 @@
#include <linux/syscalls.h>
#include <linux/slab.h>
#include <linux/anon_inodes.h>
+#include <linux/file.h>
static LIST_HEAD(bpf_map_types);
@@ -111,6 +112,228 @@ free_map:
return err;
}
+/* if error is returned, fd is released.
+ * On success caller should complete fd access with matching fdput()
+ */
+struct bpf_map *bpf_map_get(struct fd f)
+{
+ struct bpf_map *map;
+
+ if (!f.file)
+ return ERR_PTR(-EBADF);
+
+ if (f.file->f_op != &bpf_map_fops) {
+ fdput(f);
+ return ERR_PTR(-EINVAL);
+ }
+
+ map = f.file->private_data;
+
+ return map;
+}
+
+/* helper to convert user pointers passed inside __aligned_u64 fields */
+static void __user *u64_to_ptr(__u64 val)
+{
+ return (void __user *) (unsigned long) val;
+}
+
+/* last field in 'union bpf_attr' used by this command */
+#define BPF_MAP_LOOKUP_ELEM_LAST_FIELD value
+
+static int map_lookup_elem(union bpf_attr *attr)
+{
+ void __user *ukey = u64_to_ptr(attr->key);
+ void __user *uvalue = u64_to_ptr(attr->value);
+ int ufd = attr->map_fd;
+ struct fd f = fdget(ufd);
+ struct bpf_map *map;
+ void *key, *value;
+ int err;
+
+ if (CHECK_ATTR(BPF_MAP_LOOKUP_ELEM))
+ return -EINVAL;
+
+ map = bpf_map_get(f);
+ if (IS_ERR(map))
+ return PTR_ERR(map);
+
+ err = -ENOMEM;
+ key = kmalloc(map->key_size, GFP_USER);
+ if (!key)
+ goto err_put;
+
+ err = -EFAULT;
+ if (copy_from_user(key, ukey, map->key_size) != 0)
+ goto free_key;
+
+ err = -ESRCH;
+ rcu_read_lock();
+ value = map->ops->map_lookup_elem(map, key);
+ if (!value)
+ goto err_unlock;
+
+ err = -EFAULT;
+ if (copy_to_user(uvalue, value, map->value_size) != 0)
+ goto err_unlock;
+
+ err = 0;
+
+err_unlock:
+ rcu_read_unlock();
+free_key:
+ kfree(key);
+err_put:
+ fdput(f);
+ return err;
+}
+
+#define BPF_MAP_UPDATE_ELEM_LAST_FIELD value
+
+static int map_update_elem(union bpf_attr *attr)
+{
+ void __user *ukey = u64_to_ptr(attr->key);
+ void __user *uvalue = u64_to_ptr(attr->value);
+ int ufd = attr->map_fd;
+ struct fd f = fdget(ufd);
+ struct bpf_map *map;
+ void *key, *value;
+ int err;
+
+ if (CHECK_ATTR(BPF_MAP_UPDATE_ELEM))
+ return -EINVAL;
+
+ map = bpf_map_get(f);
+ if (IS_ERR(map))
+ return PTR_ERR(map);
+
+ err = -ENOMEM;
+ key = kmalloc(map->key_size, GFP_USER);
+ if (!key)
+ goto err_put;
+
+ err = -EFAULT;
+ if (copy_from_user(key, ukey, map->key_size) != 0)
+ goto free_key;
+
+ err = -ENOMEM;
+ value = kmalloc(map->value_size, GFP_USER);
+ if (!value)
+ goto free_key;
+
+ err = -EFAULT;
+ if (copy_from_user(value, uvalue, map->value_size) != 0)
+ goto free_value;
+
+ /* eBPF program that use maps are running under rcu_read_lock(),
+ * therefore all map accessors rely on this fact, so do the same here
+ */
+ rcu_read_lock();
+ err = map->ops->map_update_elem(map, key, value);
+ rcu_read_unlock();
+
+free_value:
+ kfree(value);
+free_key:
+ kfree(key);
+err_put:
+ fdput(f);
+ return err;
+}
+
+#define BPF_MAP_DELETE_ELEM_LAST_FIELD key
+
+static int map_delete_elem(union bpf_attr *attr)
+{
+ void __user *ukey = u64_to_ptr(attr->key);
+ int ufd = attr->map_fd;
+ struct fd f = fdget(ufd);
+ struct bpf_map *map;
+ void *key;
+ int err;
+
+ if (CHECK_ATTR(BPF_MAP_DELETE_ELEM))
+ return -EINVAL;
+
+ map = bpf_map_get(f);
+ if (IS_ERR(map))
+ return PTR_ERR(map);
+
+ err = -ENOMEM;
+ key = kmalloc(map->key_size, GFP_USER);
+ if (!key)
+ goto err_put;
+
+ err = -EFAULT;
+ if (copy_from_user(key, ukey, map->key_size) != 0)
+ goto free_key;
+
+ rcu_read_lock();
+ err = map->ops->map_delete_elem(map, key);
+ rcu_read_unlock();
+
+free_key:
+ kfree(key);
+err_put:
+ fdput(f);
+ return err;
+}
+
+/* last field in 'union bpf_attr' used by this command */
+#define BPF_MAP_GET_NEXT_KEY_LAST_FIELD next_key
+
+static int map_get_next_key(union bpf_attr *attr)
+{
+ void __user *ukey = u64_to_ptr(attr->key);
+ void __user *unext_key = u64_to_ptr(attr->next_key);
+ int ufd = attr->map_fd;
+ struct fd f = fdget(ufd);
+ struct bpf_map *map;
+ void *key, *next_key;
+ int err;
+
+ if (CHECK_ATTR(BPF_MAP_GET_NEXT_KEY))
+ return -EINVAL;
+
+ map = bpf_map_get(f);
+ if (IS_ERR(map))
+ return PTR_ERR(map);
+
+ err = -ENOMEM;
+ key = kmalloc(map->key_size, GFP_USER);
+ if (!key)
+ goto err_put;
+
+ err = -EFAULT;
+ if (copy_from_user(key, ukey, map->key_size) != 0)
+ goto free_key;
+
+ err = -ENOMEM;
+ next_key = kmalloc(map->key_size, GFP_USER);
+ if (!next_key)
+ goto free_key;
+
+ rcu_read_lock();
+ err = map->ops->map_get_next_key(map, key, next_key);
+ rcu_read_unlock();
+ if (err)
+ goto free_next_key;
+
+ err = -EFAULT;
+ if (copy_to_user(unext_key, next_key, map->key_size) != 0)
+ goto free_next_key;
+
+ err = 0;
+
+free_next_key:
+ kfree(next_key);
+free_key:
+ kfree(key);
+err_put:
+ fdput(f);
+ return err;
+}
+
SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
{
union bpf_attr attr = {};
@@ -160,6 +383,18 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
case BPF_MAP_CREATE:
err = map_create(&attr);
break;
+ case BPF_MAP_LOOKUP_ELEM:
+ err = map_lookup_elem(&attr);
+ break;
+ case BPF_MAP_UPDATE_ELEM:
+ err = map_update_elem(&attr);
+ break;
+ case BPF_MAP_DELETE_ELEM:
+ err = map_delete_elem(&attr);
+ break;
+ case BPF_MAP_GET_NEXT_KEY:
+ err = map_get_next_key(&attr);
+ break;
default:
err = -EINVAL;
break;
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 04/11] bpf: expand BPF syscall with program load/unload
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
eBPF programs are similar to kernel modules. They are loaded by the user
process and automatically unloaded when process exits. Each eBPF program is
a safe run-to-completion set of instructions. eBPF verifier statically
determines that the program terminates and is safe to execute.
The following syscall wrapper can be used to load the program:
int bpf_prog_load(enum bpf_prog_type prog_type,
const struct bpf_insn *insns, int insn_cnt,
const char *license)
{
union bpf_attr attr = {
.prog_type = prog_type,
.insns = ptr_to_u64(insns),
.insn_cnt = insn_cnt,
.license = ptr_to_u64(license),
};
return bpf(BPF_PROG_LOAD, &attr, sizeof(attr));
}
where 'insns' is an array of eBPF instructions and 'license' is a string
that must be GPL compatible to call helper functions marked gpl_only
Upon succesful load the syscall returns prog_fd.
Use close(prog_fd) to unload the program.
User space tests and examples follow in the later patches
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
include/linux/bpf.h | 38 +++++++++++
include/linux/filter.h | 8 +--
include/uapi/linux/bpf.h | 26 ++++++++
kernel/bpf/core.c | 29 ++++----
kernel/bpf/syscall.c | 165 ++++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 246 insertions(+), 20 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 2887f3f9da59..92979182be81 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -46,4 +46,42 @@ void bpf_register_map_type(struct bpf_map_type_list *tl);
void bpf_map_put(struct bpf_map *map);
struct bpf_map *bpf_map_get(struct fd f);
+/* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs
+ * to in-kernel helper functions and for adjusting imm32 field in BPF_CALL
+ * instructions after verifying
+ */
+struct bpf_func_proto {
+ u64 (*func)(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
+ bool gpl_only;
+};
+
+struct bpf_verifier_ops {
+ /* return eBPF function prototype for verification */
+ const struct bpf_func_proto *(*get_func_proto)(enum bpf_func_id func_id);
+};
+
+struct bpf_prog_type_list {
+ struct list_head list_node;
+ struct bpf_verifier_ops *ops;
+ enum bpf_prog_type type;
+};
+
+void bpf_register_prog_type(struct bpf_prog_type_list *tl);
+
+struct bpf_prog;
+
+struct bpf_prog_aux {
+ atomic_t refcnt;
+ bool is_gpl_compatible;
+ enum bpf_prog_type prog_type;
+ struct bpf_verifier_ops *ops;
+ struct bpf_map **used_maps;
+ u32 used_map_cnt;
+ struct bpf_prog *prog;
+ struct work_struct work;
+};
+
+void bpf_prog_put(struct bpf_prog *prog);
+struct bpf_prog *bpf_prog_get(u32 ufd);
+
#endif /* _LINUX_BPF_H */
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 1a0bc6d134d7..4ffc0958d85e 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -21,6 +21,7 @@
struct sk_buff;
struct sock;
struct seccomp_data;
+struct bpf_prog_aux;
/* ArgX, context and stack frame pointer register positions. Note,
* Arg1, Arg2, Arg3, etc are used as argument mappings of function
@@ -300,17 +301,12 @@ struct bpf_binary_header {
u8 image[];
};
-struct bpf_work_struct {
- struct bpf_prog *prog;
- struct work_struct work;
-};
-
struct bpf_prog {
u16 pages; /* Number of allocated pages */
bool jited; /* Is our filter JIT'ed? */
u32 len; /* Number of filter blocks */
struct sock_fprog_kern *orig_prog; /* Original BPF program */
- struct bpf_work_struct *work; /* Deferred free work struct */
+ struct bpf_prog_aux *aux; /* Auxiliary fields */
unsigned int (*bpf_func)(const struct sk_buff *skb,
const struct bpf_insn *filter);
/* Instructions for interpreter */
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 395cabd2ca0a..424f442016e7 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -99,12 +99,23 @@ enum bpf_cmd {
* returns zero and stores next key or negative error
*/
BPF_MAP_GET_NEXT_KEY,
+
+ /* verify and load eBPF program
+ * prog_fd = bpf(BPF_PROG_LOAD, union bpf_attr *attr, u32 size)
+ * Using attr->prog_type, attr->insns, attr->license
+ * returns fd or negative error
+ */
+ BPF_PROG_LOAD,
};
enum bpf_map_type {
BPF_MAP_TYPE_UNSPEC,
};
+enum bpf_prog_type {
+ BPF_PROG_TYPE_UNSPEC,
+};
+
union bpf_attr {
struct { /* anonymous struct used by BPF_MAP_CREATE command */
__u32 map_type; /* one of enum bpf_map_type */
@@ -121,6 +132,21 @@ union bpf_attr {
__aligned_u64 next_key;
};
};
+
+ struct { /* anonymous struct used by BPF_PROG_LOAD command */
+ __u32 prog_type; /* one of enum bpf_prog_type */
+ __u32 insn_cnt;
+ __aligned_u64 insns;
+ __aligned_u64 license;
+ };
} __attribute__((aligned(8)));
+/* integer value in 'imm' field of BPF_CALL instruction selects which helper
+ * function eBPF program intends to call
+ */
+enum bpf_func_id {
+ BPF_FUNC_unspec,
+ __BPF_FUNC_MAX_ID,
+};
+
#endif /* _UAPI__LINUX_BPF_H__ */
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 8b7002488251..f0c30c59b317 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -27,6 +27,7 @@
#include <linux/random.h>
#include <linux/moduleloader.h>
#include <asm/unaligned.h>
+#include <linux/bpf.h>
/* Registers */
#define BPF_R0 regs[BPF_REG_0]
@@ -71,7 +72,7 @@ struct bpf_prog *bpf_prog_alloc(unsigned int size, gfp_t gfp_extra_flags)
{
gfp_t gfp_flags = GFP_KERNEL | __GFP_HIGHMEM | __GFP_ZERO |
gfp_extra_flags;
- struct bpf_work_struct *ws;
+ struct bpf_prog_aux *aux;
struct bpf_prog *fp;
size = round_up(size, PAGE_SIZE);
@@ -79,14 +80,14 @@ struct bpf_prog *bpf_prog_alloc(unsigned int size, gfp_t gfp_extra_flags)
if (fp == NULL)
return NULL;
- ws = kmalloc(sizeof(*ws), GFP_KERNEL | gfp_extra_flags);
- if (ws == NULL) {
+ aux = kzalloc(sizeof(*aux), GFP_KERNEL | gfp_extra_flags);
+ if (aux == NULL) {
vfree(fp);
return NULL;
}
fp->pages = size / PAGE_SIZE;
- fp->work = ws;
+ fp->aux = aux;
return fp;
}
@@ -110,10 +111,10 @@ struct bpf_prog *bpf_prog_realloc(struct bpf_prog *fp_old, unsigned int size,
memcpy(fp, fp_old, fp_old->pages * PAGE_SIZE);
fp->pages = size / PAGE_SIZE;
- /* We keep fp->work from fp_old around in the new
+ /* We keep fp->aux from fp_old around in the new
* reallocated structure.
*/
- fp_old->work = NULL;
+ fp_old->aux = NULL;
__bpf_prog_free(fp_old);
}
@@ -123,7 +124,7 @@ EXPORT_SYMBOL_GPL(bpf_prog_realloc);
void __bpf_prog_free(struct bpf_prog *fp)
{
- kfree(fp->work);
+ kfree(fp->aux);
vfree(fp);
}
EXPORT_SYMBOL_GPL(__bpf_prog_free);
@@ -638,19 +639,19 @@ EXPORT_SYMBOL_GPL(bpf_prog_select_runtime);
static void bpf_prog_free_deferred(struct work_struct *work)
{
- struct bpf_work_struct *ws;
+ struct bpf_prog_aux *aux;
- ws = container_of(work, struct bpf_work_struct, work);
- bpf_jit_free(ws->prog);
+ aux = container_of(work, struct bpf_prog_aux, work);
+ bpf_jit_free(aux->prog);
}
/* Free internal BPF program */
void bpf_prog_free(struct bpf_prog *fp)
{
- struct bpf_work_struct *ws = fp->work;
+ struct bpf_prog_aux *aux = fp->aux;
- INIT_WORK(&ws->work, bpf_prog_free_deferred);
- ws->prog = fp;
- schedule_work(&ws->work);
+ INIT_WORK(&aux->work, bpf_prog_free_deferred);
+ aux->prog = fp;
+ schedule_work(&aux->work);
}
EXPORT_SYMBOL_GPL(bpf_prog_free);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index f94349ecaf61..0afb4eaa1887 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -14,6 +14,8 @@
#include <linux/slab.h>
#include <linux/anon_inodes.h>
#include <linux/file.h>
+#include <linux/license.h>
+#include <linux/filter.h>
static LIST_HEAD(bpf_map_types);
@@ -334,6 +336,166 @@ err_put:
return err;
}
+static LIST_HEAD(bpf_prog_types);
+
+static int find_prog_type(enum bpf_prog_type type, struct bpf_prog *prog)
+{
+ struct bpf_prog_type_list *tl;
+
+ list_for_each_entry(tl, &bpf_prog_types, list_node) {
+ if (tl->type == type) {
+ prog->aux->ops = tl->ops;
+ prog->aux->prog_type = type;
+ return 0;
+ }
+ }
+ return -EINVAL;
+}
+
+void bpf_register_prog_type(struct bpf_prog_type_list *tl)
+{
+ list_add(&tl->list_node, &bpf_prog_types);
+}
+
+/* drop refcnt on maps used by eBPF program and free auxilary data */
+static void free_used_maps(struct bpf_prog_aux *aux)
+{
+ int i;
+
+ for (i = 0; i < aux->used_map_cnt; i++)
+ bpf_map_put(aux->used_maps[i]);
+
+ kfree(aux->used_maps);
+}
+
+void bpf_prog_put(struct bpf_prog *prog)
+{
+ if (atomic_dec_and_test(&prog->aux->refcnt)) {
+ free_used_maps(prog->aux);
+ bpf_prog_free(prog);
+ }
+}
+
+static int bpf_prog_release(struct inode *inode, struct file *filp)
+{
+ struct bpf_prog *prog = filp->private_data;
+
+ bpf_prog_put(prog);
+ return 0;
+}
+
+static const struct file_operations bpf_prog_fops = {
+ .release = bpf_prog_release,
+};
+
+static struct bpf_prog *get_prog(struct fd f)
+{
+ struct bpf_prog *prog;
+
+ if (!f.file)
+ return ERR_PTR(-EBADF);
+
+ if (f.file->f_op != &bpf_prog_fops) {
+ fdput(f);
+ return ERR_PTR(-EINVAL);
+ }
+
+ prog = f.file->private_data;
+
+ return prog;
+}
+
+/* called by sockets/tracing/seccomp before attaching program to an event
+ * pairs with bpf_prog_put()
+ */
+struct bpf_prog *bpf_prog_get(u32 ufd)
+{
+ struct fd f = fdget(ufd);
+ struct bpf_prog *prog;
+
+ prog = get_prog(f);
+
+ if (IS_ERR(prog))
+ return prog;
+
+ atomic_inc(&prog->aux->refcnt);
+ fdput(f);
+ return prog;
+}
+
+/* last field in 'union bpf_attr' used by this command */
+#define BPF_PROG_LOAD_LAST_FIELD license
+
+static int bpf_prog_load(union bpf_attr *attr)
+{
+ enum bpf_prog_type type = attr->prog_type;
+ struct bpf_prog *prog;
+ int err;
+ char license[128];
+ bool is_gpl;
+
+ if (CHECK_ATTR(BPF_PROG_LOAD))
+ return -EINVAL;
+
+ /* copy eBPF program license from user space */
+ if (strncpy_from_user(license, u64_to_ptr(attr->license),
+ sizeof(license) - 1) < 0)
+ return -EFAULT;
+ license[sizeof(license) - 1] = 0;
+
+ /* eBPF programs must be GPL compatible to use GPL-ed functions */
+ is_gpl = license_is_gpl_compatible(license);
+
+ if (attr->insn_cnt >= BPF_MAXINSNS)
+ return -EINVAL;
+
+ /* plain bpf_prog allocation */
+ prog = bpf_prog_alloc(bpf_prog_size(attr->insn_cnt), GFP_USER);
+ if (!prog)
+ return -ENOMEM;
+
+ prog->len = attr->insn_cnt;
+
+ err = -EFAULT;
+ if (copy_from_user(prog->insns, u64_to_ptr(attr->insns),
+ prog->len * sizeof(struct bpf_insn)) != 0)
+ goto free_prog;
+
+ prog->orig_prog = NULL;
+ prog->jited = false;
+
+ atomic_set(&prog->aux->refcnt, 1);
+ prog->aux->is_gpl_compatible = is_gpl;
+
+ /* find program type: socket_filter vs tracing_filter */
+ err = find_prog_type(type, prog);
+ if (err < 0)
+ goto free_prog;
+
+ /* run eBPF verifier */
+ /* err = bpf_check(prog, tb); */
+
+ if (err < 0)
+ goto free_used_maps;
+
+ /* eBPF program is ready to be JITed */
+ bpf_prog_select_runtime(prog);
+
+ err = anon_inode_getfd("bpf-prog", &bpf_prog_fops, prog, O_RDWR | O_CLOEXEC);
+
+ if (err < 0)
+ /* failed to allocate fd */
+ goto free_used_maps;
+
+ return err;
+
+free_used_maps:
+ free_used_maps(prog->aux);
+free_prog:
+ bpf_prog_free(prog);
+ return err;
+}
+
SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
{
union bpf_attr attr = {};
@@ -395,6 +557,9 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
case BPF_MAP_GET_NEXT_KEY:
err = map_get_next_key(&attr);
break;
+ case BPF_PROG_LOAD:
+ err = bpf_prog_load(&attr);
+ break;
default:
err = -EINVAL;
break;
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 05/11] bpf: handle pseudo BPF_CALL insn
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
in native eBPF programs userspace is using pseudo BPF_CALL instructions
which encode one of 'enum bpf_func_id' inside insn->imm field.
Verifier checks that program using correct function arguments to given func_id.
If all checks passed, kernel needs to fixup BPF_CALL->imm fields by
replacing func_id with in-kernel function pointer.
eBPF interpreter just calls the function.
In-kernel eBPF users continue to use generic BPF_CALL.
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
kernel/bpf/syscall.c | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 0afb4eaa1887..b513659d120f 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -357,6 +357,40 @@ void bpf_register_prog_type(struct bpf_prog_type_list *tl)
list_add(&tl->list_node, &bpf_prog_types);
}
+/* fixup insn->imm field of bpf_call instructions:
+ * if (insn->imm == BPF_FUNC_map_lookup_elem)
+ * insn->imm = bpf_map_lookup_elem - __bpf_call_base;
+ * else if (insn->imm == BPF_FUNC_map_update_elem)
+ * insn->imm = bpf_map_update_elem - __bpf_call_base;
+ * else ...
+ *
+ * this function is called after eBPF program passed verification
+ */
+static void fixup_bpf_calls(struct bpf_prog *prog)
+{
+ const struct bpf_func_proto *fn;
+ int i;
+
+ for (i = 0; i < prog->len; i++) {
+ struct bpf_insn *insn = &prog->insnsi[i];
+
+ if (insn->code == (BPF_JMP | BPF_CALL)) {
+ /* we reach here when program has bpf_call instructions
+ * and it passed bpf_check(), means that
+ * ops->get_func_proto must have been supplied, check it
+ */
+ BUG_ON(!prog->aux->ops->get_func_proto);
+
+ fn = prog->aux->ops->get_func_proto(insn->imm);
+ /* all functions that have prototype and verifier allowed
+ * programs to call them, must be real in-kernel functions
+ */
+ BUG_ON(!fn->func);
+ insn->imm = fn->func - __bpf_call_base;
+ }
+ }
+}
+
/* drop refcnt on maps used by eBPF program and free auxilary data */
static void free_used_maps(struct bpf_prog_aux *aux)
{
@@ -478,6 +512,9 @@ static int bpf_prog_load(union bpf_attr *attr)
if (err < 0)
goto free_used_maps;
+ /* fixup BPF_CALL->imm field */
+ fixup_bpf_calls(prog);
+
/* eBPF program is ready to be JITed */
bpf_prog_select_runtime(prog);
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 06/11] bpf: verifier (add docs)
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
this patch adds all of eBPF verfier documentation and empty bpf_check()
The end goal for the verifier is to statically check safety of the program.
Verifier will catch:
- loops
- out of range jumps
- unreachable instructions
- invalid instructions
- uninitialized register access
- uninitialized stack access
- misaligned stack access
- out of range stack access
- invalid calling convention
More details in Documentation/networking/filter.txt
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
Documentation/networking/filter.txt | 224 +++++++++++++++++++++++++++++++++++
include/linux/bpf.h | 2 +
kernel/bpf/Makefile | 2 +-
kernel/bpf/syscall.c | 2 +-
kernel/bpf/verifier.c | 133 +++++++++++++++++++++
5 files changed, 361 insertions(+), 2 deletions(-)
create mode 100644 kernel/bpf/verifier.c
diff --git a/Documentation/networking/filter.txt b/Documentation/networking/filter.txt
index 1900d29518f1..f1c90967f748 100644
--- a/Documentation/networking/filter.txt
+++ b/Documentation/networking/filter.txt
@@ -1001,6 +1001,99 @@ instruction that loads 64-bit immediate value into a dst_reg.
Classic BPF has similar instruction: BPF_LD | BPF_W | BPF_IMM which loads
32-bit immediate value into a register.
+eBPF verifier
+-------------
+The safety of the eBPF program is determined in two steps.
+
+First step does DAG check to disallow loops and other CFG validation.
+In particular it will detect programs that have unreachable instructions.
+(though classic BPF checker allows them)
+
+Second step starts from the first insn and descends all possible paths.
+It simulates execution of every insn and observes the state change of
+registers and stack.
+
+At the start of the program the register R1 contains a pointer to context
+and has type PTR_TO_CTX.
+If verifier sees an insn that does R2=R1, then R2 has now type
+PTR_TO_CTX as well and can be used on the right hand side of expression.
+If R1=PTR_TO_CTX and insn is R2=R1+R1, then R2=UNKNOWN_VALUE,
+since addition of two valid pointers makes invalid pointer.
+(In 'secure' mode verifier will reject any type of pointer arithmetic to make
+sure that kernel addresses don't leak to unprivileged users)
+
+If register was never written to, it's not readable:
+ bpf_mov R0 = R2
+ bpf_exit
+will be rejected, since R2 is unreadable at the start of the program.
+
+After kernel function call, R1-R5 are reset to unreadable and
+R0 has a return type of the function.
+
+Since R6-R9 are callee saved, their state is preserved across the call.
+ bpf_mov R6 = 1
+ bpf_call foo
+ bpf_mov R0 = R6
+ bpf_exit
+is a correct program. If there was R1 instead of R6, it would have
+been rejected.
+
+load/store instructions are allowed only with registers of valid types, which
+are PTR_TO_CTX, PTR_TO_MAP, FRAME_PTR. They are bounds and alignment checked.
+For example:
+ bpf_mov R1 = 1
+ bpf_mov R2 = 2
+ bpf_xadd *(u32 *)(R1 + 3) += R2
+ bpf_exit
+will be rejected, since R1 doesn't have a valid pointer type at the time of
+execution of instruction bpf_xadd.
+
+At the start R1 type is PTR_TO_CTX (a pointer to generic 'struct bpf_context')
+A callback is used to customize verifier to restrict eBPF program access to only
+certain fields within ctx structure with specified size and alignment.
+
+For example, the following insn:
+ bpf_ld R0 = *(u32 *)(R6 + 8)
+intends to load a word from address R6 + 8 and store it into R0
+If R6=PTR_TO_CTX, via is_valid_access() callback the verifier will know
+that offset 8 of size 4 bytes can be accessed for reading, otherwise
+the verifier will reject the program.
+If R6=FRAME_PTR, then access should be aligned and be within
+stack bounds, which are [-MAX_BPF_STACK, 0). In this example offset is 8,
+so it will fail verification, since it's out of bounds.
+
+The verifier will allow eBPF program to read data from stack only after
+it wrote into it.
+Classic BPF verifier does similar check with M[0-15] memory slots.
+For example:
+ bpf_ld R0 = *(u32 *)(R10 - 4)
+ bpf_exit
+is invalid program.
+Though R10 is correct read-only register and has type FRAME_PTR
+and R10 - 4 is within stack bounds, there were no stores into that location.
+
+Pointer register spill/fill is tracked as well, since four (R6-R9)
+callee saved registers may not be enough for some programs.
+
+Allowed function calls are customized with bpf_verifier_ops->get_func_proto()
+The eBPF verifier will check that registers match argument constraints.
+After the call register R0 will be set to return type of the function.
+
+Function calls is a main mechanism to extend functionality of eBPF programs.
+Socket filters may let programs to call one set of functions, whereas tracing
+filters may allow completely different set.
+
+If a function made accessible to eBPF program, it needs to be thought through
+from safety point of view. The verifier will guarantee that the function is
+called with valid arguments.
+
+seccomp vs socket filters have different security restrictions for classic BPF.
+Seccomp solves this by two stage verifier: classic BPF verifier is followed
+by seccomp verifier. In case of eBPF one configurable verifier is shared for
+all use cases.
+
+See details of eBPF verifier in kernel/bpf/verifier.c
+
eBPF maps
---------
'maps' is a generic storage of different types for sharing data between kernel
@@ -1040,6 +1133,137 @@ The map is defined by:
. key size in bytes
. value size in bytes
+Understanding eBPF verifier messages
+------------------------------------
+
+The following are few examples of invalid eBPF programs and verifier error
+messages as seen in the log:
+
+Program with unreachable instructions:
+static struct bpf_insn prog[] = {
+ BPF_EXIT_INSN(),
+ BPF_EXIT_INSN(),
+};
+Error:
+ unreachable insn 1
+
+Program that reads uninitialized register:
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_2),
+ BPF_EXIT_INSN(),
+Error:
+ 0: (bf) r0 = r2
+ R2 !read_ok
+
+Program that doesn't initialize R0 before exiting:
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_1),
+ BPF_EXIT_INSN(),
+Error:
+ 0: (bf) r2 = r1
+ 1: (95) exit
+ R0 !read_ok
+
+Program that accesses stack out of bounds:
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, 8, 0),
+ BPF_EXIT_INSN(),
+Error:
+ 0: (7a) *(u64 *)(r10 +8) = 0
+ invalid stack off=8 size=8
+
+Program that doesn't initialize stack before passing its address into function:
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_EXIT_INSN(),
+Error:
+ 0: (bf) r2 = r10
+ 1: (07) r2 += -8
+ 2: (b7) r1 = 0x0
+ 3: (85) call 1
+ invalid indirect read from stack off -8+0 size 8
+
+Program that uses invalid map_fd=0 while calling to map_lookup_elem() function:
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_EXIT_INSN(),
+Error:
+ 0: (7a) *(u64 *)(r10 -8) = 0
+ 1: (bf) r2 = r10
+ 2: (07) r2 += -8
+ 3: (b7) r1 = 0x0
+ 4: (85) call 1
+ fd 0 is not pointing to valid bpf_map
+
+Program that doesn't check return value of map_lookup_elem() before accessing
+map element:
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0),
+ BPF_EXIT_INSN(),
+Error:
+ 0: (7a) *(u64 *)(r10 -8) = 0
+ 1: (bf) r2 = r10
+ 2: (07) r2 += -8
+ 3: (b7) r1 = 0x0
+ 4: (85) call 1
+ 5: (7a) *(u64 *)(r0 +0) = 0
+ R0 invalid mem access 'map_value_or_null'
+
+Program that correctly checks map_lookup_elem() returned value for NULL, but
+accesses the memory with incorrect alignment:
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
+ BPF_ST_MEM(BPF_DW, BPF_REG_0, 4, 0),
+ BPF_EXIT_INSN(),
+Error:
+ 0: (7a) *(u64 *)(r10 -8) = 0
+ 1: (bf) r2 = r10
+ 2: (07) r2 += -8
+ 3: (b7) r1 = 1
+ 4: (85) call 1
+ 5: (15) if r0 == 0x0 goto pc+1
+ R0=map_ptr R10=fp
+ 6: (7a) *(u64 *)(r0 +4) = 0
+ misaligned access off 4 size 8
+
+Program that correctly checks map_lookup_elem() returned value for NULL and
+accesses memory with correct alignment in one side of 'if' branch, but fails
+to do so in the other side of 'if' branch:
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
+ BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0),
+ BPF_EXIT_INSN(),
+ BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+Error:
+ 0: (7a) *(u64 *)(r10 -8) = 0
+ 1: (bf) r2 = r10
+ 2: (07) r2 += -8
+ 3: (b7) r1 = 1
+ 4: (85) call 1
+ 5: (15) if r0 == 0x0 goto pc+2
+ R0=map_ptr R10=fp
+ 6: (7a) *(u64 *)(r0 +0) = 0
+ 7: (95) exit
+
+ from 5 to 8: R0=imm0 R10=fp
+ 8: (7a) *(u64 *)(r0 +0) = 1
+ R0 invalid mem access 'imm'
+
Testing
-------
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 92979182be81..9dfeb36f8971 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -83,5 +83,7 @@ struct bpf_prog_aux {
void bpf_prog_put(struct bpf_prog *prog);
struct bpf_prog *bpf_prog_get(u32 ufd);
+/* verify correctness of eBPF program */
+int bpf_check(struct bpf_prog *fp, union bpf_attr *attr);
#endif /* _LINUX_BPF_H */
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index e9f7334ed07a..3c726b0995b7 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -1 +1 @@
-obj-y := core.o syscall.o
+obj-y := core.o syscall.o verifier.o
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index b513659d120f..74b3628c5fdb 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -507,7 +507,7 @@ static int bpf_prog_load(union bpf_attr *attr)
goto free_prog;
/* run eBPF verifier */
- /* err = bpf_check(prog, tb); */
+ err = bpf_check(prog, attr);
if (err < 0)
goto free_used_maps;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
new file mode 100644
index 000000000000..d6f9c3d6b4d7
--- /dev/null
+++ b/kernel/bpf/verifier.c
@@ -0,0 +1,133 @@
+/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/bpf.h>
+#include <linux/filter.h>
+#include <net/netlink.h>
+#include <linux/file.h>
+#include <linux/vmalloc.h>
+
+/* bpf_check() is a static code analyzer that walks eBPF program
+ * instruction by instruction and updates register/stack state.
+ * All paths of conditional branches are analyzed until 'bpf_exit' insn.
+ *
+ * The first pass is depth-first-search to check that the program is a DAG.
+ * It rejects the following programs:
+ * - larger than BPF_MAXINSNS insns
+ * - if loop is present (detected via back-edge)
+ * - unreachable insns exist (shouldn't be a forest. program = one function)
+ * - out of bounds or malformed jumps
+ * The second pass is all possible path descent from the 1st insn.
+ * Since it's analyzing all pathes through the program, the length of the
+ * analysis is limited to 32k insn, which may be hit even if total number of
+ * insn is less then 4K, but there are too many branches that change stack/regs.
+ * Number of 'branches to be analyzed' is limited to 1k
+ *
+ * On entry to each instruction, each register has a type, and the instruction
+ * changes the types of the registers depending on instruction semantics.
+ * If instruction is BPF_MOV64_REG(BPF_REG_1, BPF_REG_5), then type of R5 is
+ * copied to R1.
+ *
+ * All registers are 64-bit.
+ * R0 - return register
+ * R1-R5 argument passing registers
+ * R6-R9 callee saved registers
+ * R10 - frame pointer read-only
+ *
+ * At the start of BPF program the register R1 contains a pointer to bpf_context
+ * and has type PTR_TO_CTX.
+ *
+ * Verifier tracks arithmetic operations on pointers in case:
+ * BPF_MOV64_REG(BPF_REG_1, BPF_REG_10),
+ * BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -20),
+ * 1st insn copies R10 (which has FRAME_PTR) type into R1
+ * and 2nd arithmetic instruction is pattern matched to recognize
+ * that it wants to construct a pointer to some element within stack.
+ * So after 2nd insn, the register R1 has type PTR_TO_STACK
+ * (and -20 constant is saved for further stack bounds checking).
+ * Meaning that this reg is a pointer to stack plus known immediate constant.
+ *
+ * Most of the time the registers have UNKNOWN_VALUE type, which
+ * means the register has some value, but it's not a valid pointer.
+ * (like pointer plus pointer becomes UNKNOWN_VALUE type)
+ *
+ * When verifier sees load or store instructions the type of base register
+ * can be: PTR_TO_MAP_VALUE, PTR_TO_CTX, FRAME_PTR. These are three pointer
+ * types recognized by check_mem_access() function.
+ *
+ * PTR_TO_MAP_VALUE means that this register is pointing to 'map element value'
+ * and the range of [ptr, ptr + map's value_size) is accessible.
+ *
+ * registers used to pass values to function calls are checked against
+ * function argument constraints.
+ *
+ * ARG_PTR_TO_MAP_KEY is one of such argument constraints.
+ * It means that the register type passed to this function must be
+ * PTR_TO_STACK and it will be used inside the function as
+ * 'pointer to map element key'
+ *
+ * For example the argument constraints for bpf_map_lookup_elem():
+ * .ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
+ * .arg1_type = ARG_CONST_MAP_PTR,
+ * .arg2_type = ARG_PTR_TO_MAP_KEY,
+ *
+ * ret_type says that this function returns 'pointer to map elem value or null'
+ * function expects 1st argument to be a const pointer to 'struct bpf_map' and
+ * 2nd argument should be a pointer to stack, which will be used inside
+ * the helper function as a pointer to map element key.
+ *
+ * On the kernel side the helper function looks like:
+ * u64 bpf_map_lookup_elem(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5)
+ * {
+ * struct bpf_map *map = (struct bpf_map *) (unsigned long) r1;
+ * void *key = (void *) (unsigned long) r2;
+ * void *value;
+ *
+ * here kernel can access 'key' and 'map' pointers safely, knowing that
+ * [key, key + map->key_size) bytes are valid and were initialized on
+ * the stack of eBPF program.
+ * }
+ *
+ * Corresponding eBPF program may look like:
+ * BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), // after this insn R2 type is FRAME_PTR
+ * BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), // after this insn R2 type is PTR_TO_STACK
+ * BPF_LD_MAP_FD(BPF_REG_1, map_fd), // after this insn R1 type is CONST_PTR_TO_MAP
+ * BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ * here verifier looks at prototype of map_lookup_elem() and sees:
+ * .arg1_type == ARG_CONST_MAP_PTR and R1->type == CONST_PTR_TO_MAP, which is ok,
+ * Now verifier knows that this map has key of R1->map_ptr->key_size bytes
+ *
+ * Then .arg2_type == ARG_PTR_TO_MAP_KEY and R2->type == PTR_TO_STACK, ok so far,
+ * Now verifier checks that [R2, R2 + map's key_size) are within stack limits
+ * and were initialized prior to this call.
+ * If it's ok, then verifier allows this BPF_CALL insn and looks at
+ * .ret_type which is RET_PTR_TO_MAP_VALUE_OR_NULL, so it sets
+ * R0->type = PTR_TO_MAP_VALUE_OR_NULL which means bpf_map_lookup_elem() function
+ * returns ether pointer to map value or NULL.
+ *
+ * When type PTR_TO_MAP_VALUE_OR_NULL passes through 'if (reg != 0) goto +off'
+ * insn, the register holding that pointer in the true branch changes state to
+ * PTR_TO_MAP_VALUE and the same register changes state to CONST_IMM in the false
+ * branch. See check_cond_jmp_op().
+ *
+ * After the call R0 is set to return type of the function and registers R1-R5
+ * are set to NOT_INIT to indicate that they are no longer readable.
+ */
+
+int bpf_check(struct bpf_prog *prog, union bpf_attr *attr)
+{
+ int ret = -EINVAL;
+
+ return ret;
+}
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 07/11] bpf: verifier (add ability to receive verification log)
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
add optional attributes for BPF_PROG_LOAD syscall:
union bpf_attr {
struct {
...
__u32 log_level; /* verbosity level of eBPF verifier */
__u32 log_size; /* size of user buffer */
__aligned_u64 log_buf; /* user supplied 'char *buffer' */
};
};
when log_level > 0 the verifier will return its verification log in the user
supplied buffer 'log_buf' which can be used by program author to analyze why
verifier rejected given program.
'Understanding eBPF verifier messages' section of Documentation/networking/filter.txt
provides several examples of these messages, like the program:
BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
BPF_LD_MAP_FD(BPF_REG_1, 0),
BPF_CALL_FUNC(BPF_FUNC_map_lookup_elem),
BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
BPF_ST_MEM(BPF_DW, BPF_REG_0, 4, 0),
BPF_EXIT_INSN(),
will be rejected with the following multi-line message in log_buf:
0: (7a) *(u64 *)(r10 -8) = 0
1: (bf) r2 = r10
2: (07) r2 += -8
3: (b7) r1 = 0
4: (85) call 1
5: (15) if r0 == 0x0 goto pc+1
R0=map_ptr R10=fp
6: (7a) *(u64 *)(r0 +4) = 0
misaligned access off 4 size 8
The format of the output can change at any time as verifier evolves.
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
include/uapi/linux/bpf.h | 3 +
kernel/bpf/syscall.c | 2 +-
kernel/bpf/verifier.c | 235 ++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 239 insertions(+), 1 deletion(-)
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 424f442016e7..31b0ac208a52 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -138,6 +138,9 @@ union bpf_attr {
__u32 insn_cnt;
__aligned_u64 insns;
__aligned_u64 license;
+ __u32 log_level; /* verbosity level of verifier */
+ __u32 log_size; /* size of user buffer */
+ __aligned_u64 log_buf; /* user supplied buffer */
};
} __attribute__((aligned(8)));
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 74b3628c5fdb..ba61c8c16032 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -458,7 +458,7 @@ struct bpf_prog *bpf_prog_get(u32 ufd)
}
/* last field in 'union bpf_attr' used by this command */
-#define BPF_PROG_LOAD_LAST_FIELD license
+#define BPF_PROG_LOAD_LAST_FIELD log_buf
static int bpf_prog_load(union bpf_attr *attr)
{
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d6f9c3d6b4d7..871edc1f2e1f 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -125,9 +125,244 @@
* are set to NOT_INIT to indicate that they are no longer readable.
*/
+/* single container for all structs
+ * one verifier_env per bpf_check() call
+ */
+struct verifier_env {
+};
+
+/* verbose verifier prints what it's seeing
+ * bpf_check() is called under lock, so no race to access these global vars
+ */
+static u32 log_level, log_size, log_len;
+static char *log_buf;
+
+static DEFINE_MUTEX(bpf_verifier_lock);
+
+/* log_level controls verbosity level of eBPF verifier.
+ * verbose() is used to dump the verification trace to the log, so the user
+ * can figure out what's wrong with the program
+ */
+static void verbose(const char *fmt, ...)
+{
+ va_list args;
+
+ if (log_level == 0 || log_len >= log_size - 1)
+ return;
+
+ va_start(args, fmt);
+ log_len += vscnprintf(log_buf + log_len, log_size - log_len, fmt, args);
+ va_end(args);
+}
+
+static const char *const bpf_class_string[] = {
+ [BPF_LD] = "ld",
+ [BPF_LDX] = "ldx",
+ [BPF_ST] = "st",
+ [BPF_STX] = "stx",
+ [BPF_ALU] = "alu",
+ [BPF_JMP] = "jmp",
+ [BPF_RET] = "BUG",
+ [BPF_ALU64] = "alu64",
+};
+
+static const char *const bpf_alu_string[] = {
+ [BPF_ADD >> 4] = "+=",
+ [BPF_SUB >> 4] = "-=",
+ [BPF_MUL >> 4] = "*=",
+ [BPF_DIV >> 4] = "/=",
+ [BPF_OR >> 4] = "|=",
+ [BPF_AND >> 4] = "&=",
+ [BPF_LSH >> 4] = "<<=",
+ [BPF_RSH >> 4] = ">>=",
+ [BPF_NEG >> 4] = "neg",
+ [BPF_MOD >> 4] = "%=",
+ [BPF_XOR >> 4] = "^=",
+ [BPF_MOV >> 4] = "=",
+ [BPF_ARSH >> 4] = "s>>=",
+ [BPF_END >> 4] = "endian",
+};
+
+static const char *const bpf_ldst_string[] = {
+ [BPF_W >> 3] = "u32",
+ [BPF_H >> 3] = "u16",
+ [BPF_B >> 3] = "u8",
+ [BPF_DW >> 3] = "u64",
+};
+
+static const char *const bpf_jmp_string[] = {
+ [BPF_JA >> 4] = "jmp",
+ [BPF_JEQ >> 4] = "==",
+ [BPF_JGT >> 4] = ">",
+ [BPF_JGE >> 4] = ">=",
+ [BPF_JSET >> 4] = "&",
+ [BPF_JNE >> 4] = "!=",
+ [BPF_JSGT >> 4] = "s>",
+ [BPF_JSGE >> 4] = "s>=",
+ [BPF_CALL >> 4] = "call",
+ [BPF_EXIT >> 4] = "exit",
+};
+
+static void print_bpf_insn(struct bpf_insn *insn)
+{
+ u8 class = BPF_CLASS(insn->code);
+
+ if (class == BPF_ALU || class == BPF_ALU64) {
+ if (BPF_SRC(insn->code) == BPF_X)
+ verbose("(%02x) %sr%d %s %sr%d\n",
+ insn->code, class == BPF_ALU ? "(u32) " : "",
+ insn->dst_reg,
+ bpf_alu_string[BPF_OP(insn->code) >> 4],
+ class == BPF_ALU ? "(u32) " : "",
+ insn->src_reg);
+ else
+ verbose("(%02x) %sr%d %s %s%d\n",
+ insn->code, class == BPF_ALU ? "(u32) " : "",
+ insn->dst_reg,
+ bpf_alu_string[BPF_OP(insn->code) >> 4],
+ class == BPF_ALU ? "(u32) " : "",
+ insn->imm);
+ } else if (class == BPF_STX) {
+ if (BPF_MODE(insn->code) == BPF_MEM)
+ verbose("(%02x) *(%s *)(r%d %+d) = r%d\n",
+ insn->code,
+ bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
+ insn->dst_reg,
+ insn->off, insn->src_reg);
+ else if (BPF_MODE(insn->code) == BPF_XADD)
+ verbose("(%02x) lock *(%s *)(r%d %+d) += r%d\n",
+ insn->code,
+ bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
+ insn->dst_reg, insn->off,
+ insn->src_reg);
+ else
+ verbose("BUG_%02x\n", insn->code);
+ } else if (class == BPF_ST) {
+ if (BPF_MODE(insn->code) != BPF_MEM) {
+ verbose("BUG_st_%02x\n", insn->code);
+ return;
+ }
+ verbose("(%02x) *(%s *)(r%d %+d) = %d\n",
+ insn->code,
+ bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
+ insn->dst_reg,
+ insn->off, insn->imm);
+ } else if (class == BPF_LDX) {
+ if (BPF_MODE(insn->code) != BPF_MEM) {
+ verbose("BUG_ldx_%02x\n", insn->code);
+ return;
+ }
+ verbose("(%02x) r%d = *(%s *)(r%d %+d)\n",
+ insn->code, insn->dst_reg,
+ bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
+ insn->src_reg, insn->off);
+ } else if (class == BPF_LD) {
+ if (BPF_MODE(insn->code) == BPF_ABS) {
+ verbose("(%02x) r0 = *(%s *)skb[%d]\n",
+ insn->code,
+ bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
+ insn->imm);
+ } else if (BPF_MODE(insn->code) == BPF_IND) {
+ verbose("(%02x) r0 = *(%s *)skb[r%d + %d]\n",
+ insn->code,
+ bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
+ insn->src_reg, insn->imm);
+ } else if (BPF_MODE(insn->code) == BPF_IMM) {
+ verbose("(%02x) r%d = 0x%x\n",
+ insn->code, insn->dst_reg, insn->imm);
+ } else {
+ verbose("BUG_ld_%02x\n", insn->code);
+ return;
+ }
+ } else if (class == BPF_JMP) {
+ u8 opcode = BPF_OP(insn->code);
+
+ if (opcode == BPF_CALL) {
+ verbose("(%02x) call %d\n", insn->code, insn->imm);
+ } else if (insn->code == (BPF_JMP | BPF_JA)) {
+ verbose("(%02x) goto pc%+d\n",
+ insn->code, insn->off);
+ } else if (insn->code == (BPF_JMP | BPF_EXIT)) {
+ verbose("(%02x) exit\n", insn->code);
+ } else if (BPF_SRC(insn->code) == BPF_X) {
+ verbose("(%02x) if r%d %s r%d goto pc%+d\n",
+ insn->code, insn->dst_reg,
+ bpf_jmp_string[BPF_OP(insn->code) >> 4],
+ insn->src_reg, insn->off);
+ } else {
+ verbose("(%02x) if r%d %s 0x%x goto pc%+d\n",
+ insn->code, insn->dst_reg,
+ bpf_jmp_string[BPF_OP(insn->code) >> 4],
+ insn->imm, insn->off);
+ }
+ } else {
+ verbose("(%02x) %s\n", insn->code, bpf_class_string[class]);
+ }
+}
+
int bpf_check(struct bpf_prog *prog, union bpf_attr *attr)
{
+ char __user *log_ubuf = NULL;
+ struct verifier_env *env;
int ret = -EINVAL;
+ if (prog->len <= 0 || prog->len > BPF_MAXINSNS)
+ return -E2BIG;
+
+ /* 'struct verifier_env' can be global, but since it's not small,
+ * allocate/free it every time bpf_check() is called
+ */
+ env = kzalloc(sizeof(struct verifier_env), GFP_KERNEL);
+ if (!env)
+ return -ENOMEM;
+
+ /* grab the mutex to protect few globals used by verifier */
+ mutex_lock(&bpf_verifier_lock);
+
+ if (attr->log_level || attr->log_buf || attr->log_size) {
+ /* user requested verbose verifier output
+ * and supplied buffer to store the verification trace
+ */
+ log_level = attr->log_level;
+ log_ubuf = (char __user *) (unsigned long) attr->log_buf;
+ log_size = attr->log_size;
+ log_len = 0;
+
+ ret = -EINVAL;
+ /* log_* values have to be sane */
+ if (log_size < 128 || log_size > UINT_MAX >> 8 ||
+ log_level == 0 || log_ubuf == NULL)
+ goto free_env;
+
+ ret = -ENOMEM;
+ log_buf = vmalloc(log_size);
+ if (!log_buf)
+ goto free_env;
+ } else {
+ log_level = 0;
+ }
+
+ /* ret = do_check(env); */
+
+ if (log_level && log_len >= log_size - 1) {
+ BUG_ON(log_len >= log_size);
+ /* verifier log exceeded user supplied buffer */
+ ret = -ENOSPC;
+ /* fall through to return what was recorded */
+ }
+
+ /* copy verifier log back to user space including trailing zero */
+ if (log_level && copy_to_user(log_ubuf, log_buf, log_len + 1) != 0) {
+ ret = -EFAULT;
+ goto free_log_buf;
+ }
+
+
+free_log_buf:
+ if (log_level)
+ vfree(log_buf);
+free_env:
+ kfree(env);
+ mutex_unlock(&bpf_verifier_lock);
return ret;
}
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 08/11] bpf: handle pseudo BPF_LD_IMM64 insn
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
eBPF programs passed from userspace are using pseudo BPF_LD_IMM64 instructions
to refer to process-local map_fd. Scan the program for such instructions and
if FDs are valid, convert them to 'struct bpf_map' pointers which will be used
by verifier to check access to maps in bpf_map_lookup/update() calls.
If program passes verifier, convert pseudo BPF_LD_IMM64 into generic by dropping
BPF_PSEUDO_MAP_FD flag.
Note that eBPF interpreter is generic and knows nothing about pseudo insns.
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
include/linux/filter.h | 6 ++
kernel/bpf/verifier.c | 147 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 153 insertions(+)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 4ffc0958d85e..ca95abd2bed1 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -145,6 +145,12 @@ struct bpf_prog_aux;
.off = 0, \
.imm = ((__u64) (IMM)) >> 32 })
+#define BPF_PSEUDO_MAP_FD 1
+
+/* pseudo BPF_LD_IMM64 insn used to refer to process-local map_fd */
+#define BPF_LD_MAP_FD(DST, MAP_FD) \
+ BPF_LD_IMM64_RAW(DST, BPF_PSEUDO_MAP_FD, MAP_FD)
+
/* Short form of mov based on type, BPF_X: dst_reg = src_reg, BPF_K: dst_reg = imm32 */
#define BPF_MOV64_RAW(TYPE, DST, SRC, IMM) \
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 871edc1f2e1f..7227543e474b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -125,10 +125,15 @@
* are set to NOT_INIT to indicate that they are no longer readable.
*/
+#define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
+
/* single container for all structs
* one verifier_env per bpf_check() call
*/
struct verifier_env {
+ struct bpf_prog *prog; /* eBPF program being verified */
+ struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */
+ u32 used_map_cnt; /* number of used maps */
};
/* verbose verifier prints what it's seeing
@@ -300,6 +305,115 @@ static void print_bpf_insn(struct bpf_insn *insn)
}
}
+/* return the map pointer stored inside BPF_LD_IMM64 instruction */
+static struct bpf_map *ld_imm64_to_map_ptr(struct bpf_insn *insn)
+{
+ u64 imm64 = ((u64) (u32) insn[0].imm) | ((u64) (u32) insn[1].imm) << 32;
+
+ return (struct bpf_map *) (unsigned long) imm64;
+}
+
+/* look for pseudo eBPF instructions that access map FDs and
+ * replace them with actual map pointers
+ */
+static int replace_map_fd_with_map_ptr(struct verifier_env *env)
+{
+ struct bpf_insn *insn = env->prog->insnsi;
+ int insn_cnt = env->prog->len;
+ int i, j;
+
+ for (i = 0; i < insn_cnt; i++, insn++) {
+ if (insn[0].code == (BPF_LD | BPF_IMM | BPF_DW)) {
+ struct bpf_map *map;
+ struct fd f;
+
+ if (i == insn_cnt - 1 || insn[1].code != 0 ||
+ insn[1].dst_reg != 0 || insn[1].src_reg != 0 ||
+ insn[1].off != 0) {
+ verbose("invalid bpf_ld_imm64 insn\n");
+ return -EINVAL;
+ }
+
+ if (insn->src_reg == 0)
+ /* valid generic load 64-bit imm */
+ goto next_insn;
+
+ if (insn->src_reg != BPF_PSEUDO_MAP_FD) {
+ verbose("unrecognized bpf_ld_imm64 insn\n");
+ return -EINVAL;
+ }
+
+ f = fdget(insn->imm);
+
+ map = bpf_map_get(f);
+ if (IS_ERR(map)) {
+ verbose("fd %d is not pointing to valid bpf_map\n",
+ insn->imm);
+ fdput(f);
+ return PTR_ERR(map);
+ }
+
+ /* store map pointer inside BPF_LD_IMM64 instruction */
+ insn[0].imm = (u32) (unsigned long) map;
+ insn[1].imm = ((u64) (unsigned long) map) >> 32;
+
+ /* check whether we recorded this map already */
+ for (j = 0; j < env->used_map_cnt; j++)
+ if (env->used_maps[j] == map) {
+ fdput(f);
+ goto next_insn;
+ }
+
+ if (env->used_map_cnt >= MAX_USED_MAPS) {
+ fdput(f);
+ return -E2BIG;
+ }
+
+ /* remember this map */
+ env->used_maps[env->used_map_cnt++] = map;
+
+ /* hold the map. If the program is rejected by verifier,
+ * the map will be released by release_maps() or it
+ * will be used by the valid program until it's unloaded
+ * and all maps are released in free_bpf_prog_info()
+ */
+ atomic_inc(&map->refcnt);
+
+ fdput(f);
+next_insn:
+ insn++;
+ i++;
+ }
+ }
+
+ /* now all pseudo BPF_LD_IMM64 instructions load valid
+ * 'struct bpf_map *' into a register instead of user map_fd.
+ * These pointers will be used later by verifier to validate map access.
+ */
+ return 0;
+}
+
+/* drop refcnt of maps used by the rejected program */
+static void release_maps(struct verifier_env *env)
+{
+ int i;
+
+ for (i = 0; i < env->used_map_cnt; i++)
+ bpf_map_put(env->used_maps[i]);
+}
+
+/* convert pseudo BPF_LD_IMM64 into generic BPF_LD_IMM64 */
+static void convert_pseudo_ld_imm64(struct verifier_env *env)
+{
+ struct bpf_insn *insn = env->prog->insnsi;
+ int insn_cnt = env->prog->len;
+ int i;
+
+ for (i = 0; i < insn_cnt; i++, insn++)
+ if (insn->code == (BPF_LD | BPF_IMM | BPF_DW))
+ insn->src_reg = 0;
+}
+
int bpf_check(struct bpf_prog *prog, union bpf_attr *attr)
{
char __user *log_ubuf = NULL;
@@ -316,6 +430,8 @@ int bpf_check(struct bpf_prog *prog, union bpf_attr *attr)
if (!env)
return -ENOMEM;
+ env->prog = prog;
+
/* grab the mutex to protect few globals used by verifier */
mutex_lock(&bpf_verifier_lock);
@@ -342,8 +458,14 @@ int bpf_check(struct bpf_prog *prog, union bpf_attr *attr)
log_level = 0;
}
+ ret = replace_map_fd_with_map_ptr(env);
+ if (ret < 0)
+ goto skip_full_check;
+
/* ret = do_check(env); */
+skip_full_check:
+
if (log_level && log_len >= log_size - 1) {
BUG_ON(log_len >= log_size);
/* verifier log exceeded user supplied buffer */
@@ -357,11 +479,36 @@ int bpf_check(struct bpf_prog *prog, union bpf_attr *attr)
goto free_log_buf;
}
+ if (ret == 0 && env->used_map_cnt) {
+ /* if program passed verifier, update used_maps in bpf_prog_info */
+ prog->aux->used_maps = kmalloc_array(env->used_map_cnt,
+ sizeof(env->used_maps[0]),
+ GFP_KERNEL);
+
+ if (!prog->aux->used_maps) {
+ ret = -ENOMEM;
+ goto free_log_buf;
+ }
+
+ memcpy(prog->aux->used_maps, env->used_maps,
+ sizeof(env->used_maps[0]) * env->used_map_cnt);
+ prog->aux->used_map_cnt = env->used_map_cnt;
+
+ /* program is valid. Convert pseudo bpf_ld_imm64 into generic
+ * bpf_ld_imm64 instructions
+ */
+ convert_pseudo_ld_imm64(env);
+ }
free_log_buf:
if (log_level)
vfree(log_buf);
free_env:
+ if (!prog->aux->used_maps)
+ /* if we didn't copy map pointers into bpf_prog_info, release
+ * them now. Otherwise free_bpf_prog_info() will release them.
+ */
+ release_maps(env);
kfree(env);
mutex_unlock(&bpf_verifier_lock);
return ret;
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 09/11] bpf: verifier (add branch/goto checks)
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
check that control flow graph of eBPF program is a directed acyclic graph
check_cfg() does:
- detect loops
- detect unreachable instructions
- check that program terminates with BPF_EXIT insn
- check that all branches are within program boundary
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
kernel/bpf/verifier.c | 183 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 183 insertions(+)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 7227543e474b..effab7d1c7e8 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -313,6 +313,185 @@ static struct bpf_map *ld_imm64_to_map_ptr(struct bpf_insn *insn)
return (struct bpf_map *) (unsigned long) imm64;
}
+/* non-recursive DFS pseudo code
+ * 1 procedure DFS-iterative(G,v):
+ * 2 label v as discovered
+ * 3 let S be a stack
+ * 4 S.push(v)
+ * 5 while S is not empty
+ * 6 t <- S.pop()
+ * 7 if t is what we're looking for:
+ * 8 return t
+ * 9 for all edges e in G.adjacentEdges(t) do
+ * 10 if edge e is already labelled
+ * 11 continue with the next edge
+ * 12 w <- G.adjacentVertex(t,e)
+ * 13 if vertex w is not discovered and not explored
+ * 14 label e as tree-edge
+ * 15 label w as discovered
+ * 16 S.push(w)
+ * 17 continue at 5
+ * 18 else if vertex w is discovered
+ * 19 label e as back-edge
+ * 20 else
+ * 21 // vertex w is explored
+ * 22 label e as forward- or cross-edge
+ * 23 label t as explored
+ * 24 S.pop()
+ *
+ * convention:
+ * 0x10 - discovered
+ * 0x11 - discovered and fall-through edge labelled
+ * 0x12 - discovered and fall-through and branch edges labelled
+ * 0x20 - explored
+ */
+
+enum {
+ DISCOVERED = 0x10,
+ EXPLORED = 0x20,
+ FALLTHROUGH = 1,
+ BRANCH = 2,
+};
+
+#define PUSH_INT(I) \
+ do { \
+ if (cur_stack >= insn_cnt) { \
+ ret = -E2BIG; \
+ goto free_st; \
+ } \
+ stack[cur_stack++] = I; \
+ } while (0)
+
+#define PEEK_INT() \
+ ({ \
+ int _ret; \
+ if (cur_stack == 0) \
+ _ret = -1; \
+ else \
+ _ret = stack[cur_stack - 1]; \
+ _ret; \
+ })
+
+#define POP_INT() \
+ ({ \
+ int _ret; \
+ if (cur_stack == 0) \
+ _ret = -1; \
+ else \
+ _ret = stack[--cur_stack]; \
+ _ret; \
+ })
+
+#define PUSH_INSN(T, W, E) \
+ do { \
+ int w = W; \
+ if (E == FALLTHROUGH && st[T] >= (DISCOVERED | FALLTHROUGH)) \
+ break; \
+ if (E == BRANCH && st[T] >= (DISCOVERED | BRANCH)) \
+ break; \
+ if (w < 0 || w >= insn_cnt) { \
+ verbose("jump out of range from insn %d to %d\n", T, w); \
+ ret = -EINVAL; \
+ goto free_st; \
+ } \
+ if (st[w] == 0) { \
+ /* tree-edge */ \
+ st[T] = DISCOVERED | E; \
+ st[w] = DISCOVERED; \
+ PUSH_INT(w); \
+ goto peek_stack; \
+ } else if ((st[w] & 0xF0) == DISCOVERED) { \
+ verbose("back-edge from insn %d to %d\n", T, w); \
+ ret = -EINVAL; \
+ goto free_st; \
+ } else if (st[w] == EXPLORED) { \
+ /* forward- or cross-edge */ \
+ st[T] = DISCOVERED | E; \
+ } else { \
+ verbose("insn state internal bug\n"); \
+ ret = -EFAULT; \
+ goto free_st; \
+ } \
+ } while (0)
+
+/* non-recursive depth-first-search to detect loops in BPF program
+ * loop == back-edge in directed graph
+ */
+static int check_cfg(struct verifier_env *env)
+{
+ struct bpf_insn *insns = env->prog->insnsi;
+ int insn_cnt = env->prog->len;
+ int cur_stack = 0;
+ int *stack;
+ int ret = 0;
+ int *st;
+ int i, t;
+
+ st = kzalloc(sizeof(int) * insn_cnt, GFP_KERNEL);
+ if (!st)
+ return -ENOMEM;
+
+ stack = kzalloc(sizeof(int) * insn_cnt, GFP_KERNEL);
+ if (!stack) {
+ kfree(st);
+ return -ENOMEM;
+ }
+
+ st[0] = DISCOVERED; /* mark 1st insn as discovered */
+ PUSH_INT(0);
+
+peek_stack:
+ while ((t = PEEK_INT()) != -1) {
+ if (BPF_CLASS(insns[t].code) == BPF_JMP) {
+ u8 opcode = BPF_OP(insns[t].code);
+
+ if (opcode == BPF_EXIT) {
+ goto mark_explored;
+ } else if (opcode == BPF_CALL) {
+ PUSH_INSN(t, t + 1, FALLTHROUGH);
+ } else if (opcode == BPF_JA) {
+ if (BPF_SRC(insns[t].code) != BPF_K) {
+ ret = -EINVAL;
+ goto free_st;
+ }
+ /* unconditional jump with single edge */
+ PUSH_INSN(t, t + insns[t].off + 1, FALLTHROUGH);
+ } else {
+ /* conditional jump with two edges */
+ PUSH_INSN(t, t + 1, FALLTHROUGH);
+ PUSH_INSN(t, t + insns[t].off + 1, BRANCH);
+ }
+ } else {
+ /* all other non-branch instructions with single
+ * fall-through edge
+ */
+ PUSH_INSN(t, t + 1, FALLTHROUGH);
+ }
+
+mark_explored:
+ st[t] = EXPLORED;
+ if (POP_INT() == -1) {
+ verbose("pop_int internal bug\n");
+ ret = -EFAULT;
+ goto free_st;
+ }
+ }
+
+
+ for (i = 0; i < insn_cnt; i++) {
+ if (st[i] != EXPLORED) {
+ verbose("unreachable insn %d\n", i);
+ ret = -EINVAL;
+ goto free_st;
+ }
+ }
+
+free_st:
+ kfree(st);
+ kfree(stack);
+ return ret;
+}
+
/* look for pseudo eBPF instructions that access map FDs and
* replace them with actual map pointers
*/
@@ -462,6 +641,10 @@ int bpf_check(struct bpf_prog *prog, union bpf_attr *attr)
if (ret < 0)
goto skip_full_check;
+ ret = check_cfg(env);
+ if (ret < 0)
+ goto skip_full_check;
+
/* ret = do_check(env); */
skip_full_check:
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 10/11] bpf: verifier (add verifier core)
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
This patch adds verifier core which simulates execution of every insn and
records the state of registers and program stack. Every branch instruction seen
during simulation is pushed into state stack. When verifier reaches BPF_EXIT,
it pops the state from the stack and continues until it reaches BPF_EXIT again.
For program:
1: bpf_mov r1, xxx
2: if (r1 == 0) goto 5
3: bpf_mov r0, 1
4: goto 6
5: bpf_mov r0, 2
6: bpf_exit
The verifier will walk insns: 1, 2, 3, 4, 6
then it will pop the state recorded at insn#2 and will continue: 5, 6
This way it walks all possible paths through the program and checks all
possible values of registers. While doing so, it checks for:
- invalid instructions
- uninitialized register access
- uninitialized stack access
- misaligned stack access
- out of range stack access
- invalid calling convention
- instruction encoding is not using reserved fields
Kernel subsystem configures the verifier with two callbacks:
- bool (*is_valid_access)(int off, int size, enum bpf_access_type type);
that provides information to the verifer which fields of 'ctx'
are accessible (remember 'ctx' is the first argument to eBPF program)
- const struct bpf_func_proto *(*get_func_proto)(enum bpf_func_id func_id);
returns argument constraints of kernel helper functions that eBPF program
may call, so that verifier can checks that R1-R5 types match the prototype
More details in Documentation/networking/filter.txt and in kernel/bpf/verifier.c
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
include/linux/bpf.h | 47 +++
kernel/bpf/verifier.c | 1003 ++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 1049 insertions(+), 1 deletion(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 9dfeb36f8971..3cf91754a957 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -46,6 +46,31 @@ void bpf_register_map_type(struct bpf_map_type_list *tl);
void bpf_map_put(struct bpf_map *map);
struct bpf_map *bpf_map_get(struct fd f);
+/* function argument constraints */
+enum bpf_arg_type {
+ ARG_ANYTHING = 0, /* any argument is ok */
+
+ /* the following constraints used to prototype
+ * bpf_map_lookup/update/delete_elem() functions
+ */
+ ARG_CONST_MAP_PTR, /* const argument used as pointer to bpf_map */
+ ARG_PTR_TO_MAP_KEY, /* pointer to stack used as map key */
+ ARG_PTR_TO_MAP_VALUE, /* pointer to stack used as map value */
+
+ /* the following constraints used to prototype bpf_memcmp() and other
+ * functions that access data on eBPF program stack
+ */
+ ARG_PTR_TO_STACK, /* any pointer to eBPF program stack */
+ ARG_CONST_STACK_SIZE, /* number of bytes accessed from stack */
+};
+
+/* type of values returned from helper functions */
+enum bpf_return_type {
+ RET_INTEGER, /* function returns integer */
+ RET_VOID, /* function doesn't return anything */
+ RET_PTR_TO_MAP_VALUE_OR_NULL, /* returns a pointer to map elem value or NULL */
+};
+
/* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs
* to in-kernel helper functions and for adjusting imm32 field in BPF_CALL
* instructions after verifying
@@ -53,11 +78,33 @@ struct bpf_map *bpf_map_get(struct fd f);
struct bpf_func_proto {
u64 (*func)(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
bool gpl_only;
+ enum bpf_return_type ret_type;
+ enum bpf_arg_type arg1_type;
+ enum bpf_arg_type arg2_type;
+ enum bpf_arg_type arg3_type;
+ enum bpf_arg_type arg4_type;
+ enum bpf_arg_type arg5_type;
+};
+
+/* bpf_context is intentionally undefined structure. Pointer to bpf_context is
+ * the first argument to eBPF programs.
+ * For socket filters: 'struct bpf_context *' == 'struct sk_buff *'
+ */
+struct bpf_context;
+
+enum bpf_access_type {
+ BPF_READ = 1,
+ BPF_WRITE = 2
};
struct bpf_verifier_ops {
/* return eBPF function prototype for verification */
const struct bpf_func_proto *(*get_func_proto)(enum bpf_func_id func_id);
+
+ /* return true if 'size' wide access at offset 'off' within bpf_context
+ * with 'type' (read or write) is allowed
+ */
+ bool (*is_valid_access)(int off, int size, enum bpf_access_type type);
};
struct bpf_prog_type_list {
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index effab7d1c7e8..663d02c6e23c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -125,6 +125,72 @@
* are set to NOT_INIT to indicate that they are no longer readable.
*/
+#define _(OP) ({ int ret = (OP); if (ret < 0) return ret; })
+
+/* types of values stored in eBPF registers */
+enum bpf_reg_type {
+ NOT_INIT = 0, /* nothing was written into register */
+ UNKNOWN_VALUE, /* reg doesn't contain a valid pointer */
+ PTR_TO_CTX, /* reg points to bpf_context */
+ CONST_PTR_TO_MAP, /* reg points to struct bpf_map */
+ PTR_TO_MAP_VALUE, /* reg points to map element value */
+ PTR_TO_MAP_VALUE_OR_NULL,/* points to map elem value or NULL */
+ FRAME_PTR, /* reg == frame_pointer */
+ PTR_TO_STACK, /* reg == frame_pointer + imm */
+ CONST_IMM, /* constant integer value */
+};
+
+struct reg_state {
+ enum bpf_reg_type type;
+ union {
+ /* valid when type == CONST_IMM | PTR_TO_STACK */
+ int imm;
+
+ /* valid when type == CONST_PTR_TO_MAP | PTR_TO_MAP_VALUE |
+ * PTR_TO_MAP_VALUE_OR_NULL
+ */
+ struct bpf_map *map_ptr;
+ };
+};
+
+enum bpf_stack_slot_type {
+ STACK_INVALID, /* nothing was stored in this stack slot */
+ STACK_SPILL, /* 1st byte of register spilled into stack */
+ STACK_SPILL_PART, /* other 7 bytes of register spill */
+ STACK_MISC /* BPF program wrote some data into this slot */
+};
+
+struct bpf_stack_slot {
+ enum bpf_stack_slot_type stype;
+ struct reg_state reg_st;
+};
+
+/* state of the program:
+ * type of all registers and stack info
+ */
+struct verifier_state {
+ struct reg_state regs[MAX_BPF_REG];
+ struct bpf_stack_slot stack[MAX_BPF_STACK];
+};
+
+/* linked list of verifier states used to prune search */
+struct verifier_state_list {
+ struct verifier_state state;
+ struct verifier_state_list *next;
+};
+
+/* verifier_state + insn_idx are pushed to stack when branch is encountered */
+struct verifier_stack_elem {
+ /* verifer state is 'st'
+ * before processing instruction 'insn_idx'
+ * and after processing instruction 'prev_insn_idx'
+ */
+ struct verifier_state st;
+ int insn_idx;
+ int prev_insn_idx;
+ struct verifier_stack_elem *next;
+};
+
#define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
/* single container for all structs
@@ -132,6 +198,9 @@
*/
struct verifier_env {
struct bpf_prog *prog; /* eBPF program being verified */
+ struct verifier_stack_elem *head; /* stack of verifier states to be processed */
+ int stack_size; /* number of states to be processed */
+ struct verifier_state cur_state; /* current verifier state */
struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */
u32 used_map_cnt; /* number of used maps */
};
@@ -160,6 +229,45 @@ static void verbose(const char *fmt, ...)
va_end(args);
}
+/* string representation of 'enum bpf_reg_type' */
+static const char * const reg_type_str[] = {
+ [NOT_INIT] = "?",
+ [UNKNOWN_VALUE] = "inv",
+ [PTR_TO_CTX] = "ctx",
+ [CONST_PTR_TO_MAP] = "map_ptr",
+ [PTR_TO_MAP_VALUE] = "map_value",
+ [PTR_TO_MAP_VALUE_OR_NULL] = "map_value_or_null",
+ [FRAME_PTR] = "fp",
+ [PTR_TO_STACK] = "fp",
+ [CONST_IMM] = "imm",
+};
+
+static void print_verifier_state(struct verifier_env *env)
+{
+ enum bpf_reg_type t;
+ int i;
+
+ for (i = 0; i < MAX_BPF_REG; i++) {
+ t = env->cur_state.regs[i].type;
+ if (t == NOT_INIT)
+ continue;
+ verbose(" R%d=%s", i, reg_type_str[t]);
+ if (t == CONST_IMM || t == PTR_TO_STACK)
+ verbose("%d", env->cur_state.regs[i].imm);
+ else if (t == CONST_PTR_TO_MAP || t == PTR_TO_MAP_VALUE ||
+ t == PTR_TO_MAP_VALUE_OR_NULL)
+ verbose("(ks=%d,vs=%d)",
+ env->cur_state.regs[i].map_ptr->key_size,
+ env->cur_state.regs[i].map_ptr->value_size);
+ }
+ for (i = 0; i < MAX_BPF_STACK; i++) {
+ if (env->cur_state.stack[i].stype == STACK_SPILL)
+ verbose(" fp%d=%s", -MAX_BPF_STACK + i,
+ reg_type_str[env->cur_state.stack[i].reg_st.type]);
+ }
+ verbose("\n");
+}
+
static const char *const bpf_class_string[] = {
[BPF_LD] = "ld",
[BPF_LDX] = "ldx",
@@ -305,6 +413,695 @@ static void print_bpf_insn(struct bpf_insn *insn)
}
}
+static int pop_stack(struct verifier_env *env, int *prev_insn_idx)
+{
+ struct verifier_stack_elem *elem;
+ int insn_idx;
+
+ if (env->head == NULL)
+ return -1;
+
+ memcpy(&env->cur_state, &env->head->st, sizeof(env->cur_state));
+ insn_idx = env->head->insn_idx;
+ if (prev_insn_idx)
+ *prev_insn_idx = env->head->prev_insn_idx;
+ elem = env->head->next;
+ kfree(env->head);
+ env->head = elem;
+ env->stack_size--;
+ return insn_idx;
+}
+
+static struct verifier_state *push_stack(struct verifier_env *env, int insn_idx,
+ int prev_insn_idx)
+{
+ struct verifier_stack_elem *elem;
+
+ elem = kmalloc(sizeof(struct verifier_stack_elem), GFP_KERNEL);
+ if (!elem)
+ goto err;
+
+ memcpy(&elem->st, &env->cur_state, sizeof(env->cur_state));
+ elem->insn_idx = insn_idx;
+ elem->prev_insn_idx = prev_insn_idx;
+ elem->next = env->head;
+ env->head = elem;
+ env->stack_size++;
+ if (env->stack_size > 1024) {
+ verbose("BPF program is too complex\n");
+ goto err;
+ }
+ return &elem->st;
+err:
+ /* pop all elements and return */
+ while (pop_stack(env, NULL) >= 0);
+ return NULL;
+}
+
+#define CALLER_SAVED_REGS 6
+static const int caller_saved[CALLER_SAVED_REGS] = {
+ BPF_REG_0, BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5
+};
+
+static void init_reg_state(struct reg_state *regs)
+{
+ int i;
+
+ for (i = 0; i < MAX_BPF_REG; i++) {
+ regs[i].type = NOT_INIT;
+ regs[i].imm = 0;
+ regs[i].map_ptr = NULL;
+ }
+
+ /* frame pointer */
+ regs[BPF_REG_FP].type = FRAME_PTR;
+
+ /* 1st arg to a function */
+ regs[BPF_REG_1].type = PTR_TO_CTX;
+}
+
+static void mark_reg_unknown_value(struct reg_state *regs, u32 regno)
+{
+ BUG_ON(regno >= MAX_BPF_REG);
+ regs[regno].type = UNKNOWN_VALUE;
+ regs[regno].imm = 0;
+ regs[regno].map_ptr = NULL;
+}
+
+enum reg_arg_type {
+ SRC_OP, /* register is used as source operand */
+ DST_OP, /* register is used as destination operand */
+ DST_OP_NO_MARK /* same as above, check only, don't mark */
+};
+
+static int check_reg_arg(struct reg_state *regs, u32 regno,
+ enum reg_arg_type t)
+{
+ if (regno >= MAX_BPF_REG) {
+ verbose("R%d is invalid\n", regno);
+ return -EINVAL;
+ }
+
+ if (t == SRC_OP) {
+ /* check whether register used as source operand can be read */
+ if (regs[regno].type == NOT_INIT) {
+ verbose("R%d !read_ok\n", regno);
+ return -EACCES;
+ }
+ } else {
+ /* check whether register used as dest operand can be written to */
+ if (regno == BPF_REG_FP) {
+ verbose("frame pointer is read only\n");
+ return -EACCES;
+ }
+ if (t == DST_OP)
+ mark_reg_unknown_value(regs, regno);
+ }
+ return 0;
+}
+
+static int bpf_size_to_bytes(int bpf_size)
+{
+ if (bpf_size == BPF_W)
+ return 4;
+ else if (bpf_size == BPF_H)
+ return 2;
+ else if (bpf_size == BPF_B)
+ return 1;
+ else if (bpf_size == BPF_DW)
+ return 8;
+ else
+ return -EINVAL;
+}
+
+/* check_stack_read/write functions track spill/fill of registers,
+ * stack boundary and alignment are checked in check_mem_access()
+ */
+static int check_stack_write(struct verifier_state *state, int off, int size,
+ int value_regno)
+{
+ struct bpf_stack_slot *slot;
+ int i;
+
+ if (value_regno >= 0 &&
+ (state->regs[value_regno].type == PTR_TO_MAP_VALUE ||
+ state->regs[value_regno].type == PTR_TO_STACK ||
+ state->regs[value_regno].type == PTR_TO_CTX)) {
+
+ /* register containing pointer is being spilled into stack */
+ if (size != 8) {
+ verbose("invalid size of register spill\n");
+ return -EACCES;
+ }
+
+ slot = &state->stack[MAX_BPF_STACK + off];
+ slot->stype = STACK_SPILL;
+ /* save register state */
+ slot->reg_st = state->regs[value_regno];
+ for (i = 1; i < 8; i++) {
+ slot = &state->stack[MAX_BPF_STACK + off + i];
+ slot->stype = STACK_SPILL_PART;
+ slot->reg_st.type = UNKNOWN_VALUE;
+ slot->reg_st.map_ptr = NULL;
+ }
+ } else {
+
+ /* regular write of data into stack */
+ for (i = 0; i < size; i++) {
+ slot = &state->stack[MAX_BPF_STACK + off + i];
+ slot->stype = STACK_MISC;
+ slot->reg_st.type = UNKNOWN_VALUE;
+ slot->reg_st.map_ptr = NULL;
+ }
+ }
+ return 0;
+}
+
+static int check_stack_read(struct verifier_state *state, int off, int size,
+ int value_regno)
+{
+ int i;
+ struct bpf_stack_slot *slot;
+
+ slot = &state->stack[MAX_BPF_STACK + off];
+
+ if (slot->stype == STACK_SPILL) {
+ if (size != 8) {
+ verbose("invalid size of register spill\n");
+ return -EACCES;
+ }
+ for (i = 1; i < 8; i++) {
+ if (state->stack[MAX_BPF_STACK + off + i].stype !=
+ STACK_SPILL_PART) {
+ verbose("corrupted spill memory\n");
+ return -EACCES;
+ }
+ }
+
+ if (value_regno >= 0)
+ /* restore register state from stack */
+ state->regs[value_regno] = slot->reg_st;
+ return 0;
+ } else {
+ for (i = 0; i < size; i++) {
+ if (state->stack[MAX_BPF_STACK + off + i].stype !=
+ STACK_MISC) {
+ verbose("invalid read from stack off %d+%d size %d\n",
+ off, i, size);
+ return -EACCES;
+ }
+ }
+ if (value_regno >= 0)
+ /* have read misc data from the stack */
+ mark_reg_unknown_value(state->regs, value_regno);
+ return 0;
+ }
+}
+
+/* check read/write into map element returned by bpf_map_lookup_elem() */
+static int check_map_access(struct verifier_env *env, u32 regno, int off,
+ int size)
+{
+ struct bpf_map *map = env->cur_state.regs[regno].map_ptr;
+
+ if (off < 0 || off + size > map->value_size) {
+ verbose("invalid access to map value, value_size=%d off=%d size=%d\n",
+ map->value_size, off, size);
+ return -EACCES;
+ }
+ return 0;
+}
+
+/* check access to 'struct bpf_context' fields */
+static int check_ctx_access(struct verifier_env *env, int off, int size,
+ enum bpf_access_type t)
+{
+ if (env->prog->aux->ops->is_valid_access &&
+ env->prog->aux->ops->is_valid_access(off, size, t))
+ return 0;
+
+ verbose("invalid bpf_context access off=%d size=%d\n", off, size);
+ return -EACCES;
+}
+
+/* check whether memory at (regno + off) is accessible for t = (read | write)
+ * if t==write, value_regno is a register which value is stored into memory
+ * if t==read, value_regno is a register which will receive the value from memory
+ * if t==write && value_regno==-1, some unknown value is stored into memory
+ * if t==read && value_regno==-1, don't care what we read from memory
+ */
+static int check_mem_access(struct verifier_env *env, u32 regno, int off,
+ int bpf_size, enum bpf_access_type t,
+ int value_regno)
+{
+ struct verifier_state *state = &env->cur_state;
+ int size;
+
+ _(size = bpf_size_to_bytes(bpf_size));
+
+ if (off % size != 0) {
+ verbose("misaligned access off %d size %d\n", off, size);
+ return -EACCES;
+ }
+
+ if (state->regs[regno].type == PTR_TO_MAP_VALUE) {
+ _(check_map_access(env, regno, off, size));
+ if (t == BPF_READ && value_regno >= 0)
+ mark_reg_unknown_value(state->regs, value_regno);
+
+ } else if (state->regs[regno].type == PTR_TO_CTX) {
+ _(check_ctx_access(env, off, size, t));
+ if (t == BPF_READ && value_regno >= 0)
+ mark_reg_unknown_value(state->regs, value_regno);
+
+ } else if (state->regs[regno].type == FRAME_PTR) {
+ if (off >= 0 || off < -MAX_BPF_STACK) {
+ verbose("invalid stack off=%d size=%d\n", off, size);
+ return -EACCES;
+ }
+ if (t == BPF_WRITE)
+ _(check_stack_write(state, off, size, value_regno));
+ else
+ _(check_stack_read(state, off, size, value_regno));
+ } else {
+ verbose("R%d invalid mem access '%s'\n",
+ regno, reg_type_str[state->regs[regno].type]);
+ return -EACCES;
+ }
+ return 0;
+}
+
+static int check_xadd(struct verifier_env *env, struct bpf_insn *insn)
+{
+ struct reg_state *regs = env->cur_state.regs;
+
+ if ((BPF_SIZE(insn->code) != BPF_W && BPF_SIZE(insn->code) != BPF_DW) ||
+ insn->imm != 0) {
+ verbose("BPF_XADD uses reserved fields\n");
+ return -EINVAL;
+ }
+
+ /* check src1 operand */
+ _(check_reg_arg(regs, insn->src_reg, SRC_OP));
+ /* check src2 operand */
+ _(check_reg_arg(regs, insn->dst_reg, SRC_OP));
+
+ /* check whether atomic_add can read the memory */
+ _(check_mem_access(env, insn->dst_reg, insn->off,
+ BPF_SIZE(insn->code), BPF_READ, -1));
+
+ /* check whether atomic_add can write into the same memory */
+ _(check_mem_access(env, insn->dst_reg, insn->off,
+ BPF_SIZE(insn->code), BPF_WRITE, -1));
+ return 0;
+}
+
+/* when register 'regno' is passed into function that will read 'access_size'
+ * bytes from that pointer, make sure that it's within stack boundary
+ * and all elements of stack are initialized
+ */
+static int check_stack_boundary(struct verifier_env *env,
+ int regno, int access_size)
+{
+ struct verifier_state *state = &env->cur_state;
+ struct reg_state *regs = state->regs;
+ int off, i;
+
+ if (regs[regno].type != PTR_TO_STACK)
+ return -EACCES;
+
+ off = regs[regno].imm;
+ if (off >= 0 || off < -MAX_BPF_STACK || off + access_size > 0 ||
+ access_size <= 0) {
+ verbose("invalid stack type R%d off=%d access_size=%d\n",
+ regno, off, access_size);
+ return -EACCES;
+ }
+
+ for (i = 0; i < access_size; i++) {
+ if (state->stack[MAX_BPF_STACK + off + i].stype != STACK_MISC) {
+ verbose("invalid indirect read from stack off %d+%d size %d\n",
+ off, i, access_size);
+ return -EACCES;
+ }
+ }
+ return 0;
+}
+
+static int check_func_arg(struct verifier_env *env, u32 regno,
+ enum bpf_arg_type arg_type, struct bpf_map **mapp)
+{
+ struct reg_state *reg = env->cur_state.regs + regno;
+ enum bpf_reg_type expected_type;
+
+ if (arg_type == ARG_ANYTHING)
+ return 0;
+
+ if (reg->type == NOT_INIT) {
+ verbose("R%d !read_ok\n", regno);
+ return -EACCES;
+ }
+
+ if (arg_type == ARG_PTR_TO_STACK || arg_type == ARG_PTR_TO_MAP_KEY ||
+ arg_type == ARG_PTR_TO_MAP_VALUE) {
+ expected_type = PTR_TO_STACK;
+ } else if (arg_type == ARG_CONST_STACK_SIZE) {
+ expected_type = CONST_IMM;
+ } else if (arg_type == ARG_CONST_MAP_PTR) {
+ expected_type = CONST_PTR_TO_MAP;
+ } else {
+ verbose("unsupported arg_type %d\n", arg_type);
+ return -EFAULT;
+ }
+
+ if (reg->type != expected_type) {
+ verbose("R%d type=%s expected=%s\n", regno,
+ reg_type_str[reg->type], reg_type_str[expected_type]);
+ return -EACCES;
+ }
+
+ if (arg_type == ARG_CONST_MAP_PTR) {
+ /* bpf_map_xxx(map_ptr) call: remember that map_ptr */
+ *mapp = reg->map_ptr;
+
+ } else if (arg_type == ARG_PTR_TO_MAP_KEY) {
+ /* bpf_map_xxx(..., map_ptr, ..., key) call:
+ * check that [key, key + map->key_size) are within
+ * stack limits and initialized
+ */
+ if (!*mapp) {
+ /* in function declaration map_ptr must come before
+ * map_key, so that it's verified and known before
+ * we have to check map_key here. Otherwise it means
+ * that kernel subsystem misconfigured verifier
+ */
+ verbose("invalid map_ptr to access map->key\n");
+ return -EACCES;
+ }
+ _(check_stack_boundary(env, regno, (*mapp)->key_size));
+
+ } else if (arg_type == ARG_PTR_TO_MAP_VALUE) {
+ /* bpf_map_xxx(..., map_ptr, ..., value) call:
+ * check [value, value + map->value_size) validity
+ */
+ if (!*mapp) {
+ /* kernel subsystem misconfigured verifier */
+ verbose("invalid map_ptr to access map->value\n");
+ return -EACCES;
+ }
+ _(check_stack_boundary(env, regno, (*mapp)->value_size));
+
+ } else if (arg_type == ARG_CONST_STACK_SIZE) {
+ /* bpf_xxx(..., buf, len) call will access 'len' bytes
+ * from stack pointer 'buf'. Check it
+ * note: regno == len, regno - 1 == buf
+ */
+ if (regno == 0) {
+ /* kernel subsystem misconfigured verifier */
+ verbose("ARG_CONST_STACK_SIZE cannot be first argument\n");
+ return -EACCES;
+ }
+ _(check_stack_boundary(env, regno - 1, reg->imm));
+ }
+
+ return 0;
+}
+
+static int check_call(struct verifier_env *env, int func_id)
+{
+ struct verifier_state *state = &env->cur_state;
+ const struct bpf_func_proto *fn = NULL;
+ struct reg_state *regs = state->regs;
+ struct bpf_map *map = NULL;
+ struct reg_state *reg;
+ int i;
+
+ /* find function prototype */
+ if (func_id < 0 || func_id >= __BPF_FUNC_MAX_ID) {
+ verbose("invalid func %d\n", func_id);
+ return -EINVAL;
+ }
+
+ if (env->prog->aux->ops->get_func_proto)
+ fn = env->prog->aux->ops->get_func_proto(func_id);
+
+ if (!fn) {
+ verbose("unknown func %d\n", func_id);
+ return -EINVAL;
+ }
+
+ /* eBPF programs must be GPL compatible to use GPL-ed functions */
+ if (!env->prog->aux->is_gpl_compatible && fn->gpl_only) {
+ verbose("cannot call GPL only function from proprietary program\n");
+ return -EINVAL;
+ }
+
+ /* check args */
+ _(check_func_arg(env, BPF_REG_1, fn->arg1_type, &map));
+ _(check_func_arg(env, BPF_REG_2, fn->arg2_type, &map));
+ _(check_func_arg(env, BPF_REG_3, fn->arg3_type, &map));
+ _(check_func_arg(env, BPF_REG_4, fn->arg4_type, &map));
+ _(check_func_arg(env, BPF_REG_5, fn->arg5_type, &map));
+
+ /* reset caller saved regs */
+ for (i = 0; i < CALLER_SAVED_REGS; i++) {
+ reg = regs + caller_saved[i];
+ reg->type = NOT_INIT;
+ reg->imm = 0;
+ }
+
+ /* update return register */
+ if (fn->ret_type == RET_INTEGER) {
+ regs[BPF_REG_0].type = UNKNOWN_VALUE;
+ } else if (fn->ret_type == RET_VOID) {
+ regs[BPF_REG_0].type = NOT_INIT;
+ } else if (fn->ret_type == RET_PTR_TO_MAP_VALUE_OR_NULL) {
+ regs[BPF_REG_0].type = PTR_TO_MAP_VALUE_OR_NULL;
+ /* remember map_ptr, so that check_map_access()
+ * can check 'value_size' boundary of memory access
+ * to map element returned from bpf_map_lookup_elem()
+ */
+ if (map == NULL) {
+ verbose("kernel subsystem misconfigured verifier\n");
+ return -EINVAL;
+ }
+ regs[BPF_REG_0].map_ptr = map;
+ } else {
+ verbose("unknown return type %d of func %d\n",
+ fn->ret_type, func_id);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+/* check validity of 32-bit and 64-bit arithmetic operations */
+static int check_alu_op(struct reg_state *regs, struct bpf_insn *insn)
+{
+ u8 opcode = BPF_OP(insn->code);
+
+ if (opcode == BPF_END || opcode == BPF_NEG) {
+ if (opcode == BPF_NEG) {
+ if (BPF_SRC(insn->code) != 0 ||
+ insn->src_reg != BPF_REG_0 ||
+ insn->off != 0 || insn->imm != 0) {
+ verbose("BPF_NEG uses reserved fields\n");
+ return -EINVAL;
+ }
+ } else {
+ if (insn->src_reg != BPF_REG_0 || insn->off != 0 ||
+ (insn->imm != 16 && insn->imm != 32 && insn->imm != 64)) {
+ verbose("BPF_END uses reserved fields\n");
+ return -EINVAL;
+ }
+ }
+
+ /* check src operand */
+ _(check_reg_arg(regs, insn->dst_reg, SRC_OP));
+
+ /* check dest operand */
+ _(check_reg_arg(regs, insn->dst_reg, DST_OP));
+
+ } else if (opcode == BPF_MOV) {
+
+ if (BPF_SRC(insn->code) == BPF_X) {
+ if (insn->imm != 0 || insn->off != 0) {
+ verbose("BPF_MOV uses reserved fields\n");
+ return -EINVAL;
+ }
+
+ /* check src operand */
+ _(check_reg_arg(regs, insn->src_reg, SRC_OP));
+ } else {
+ if (insn->src_reg != BPF_REG_0 || insn->off != 0) {
+ verbose("BPF_MOV uses reserved fields\n");
+ return -EINVAL;
+ }
+ }
+
+ /* check dest operand */
+ _(check_reg_arg(regs, insn->dst_reg, DST_OP));
+
+ if (BPF_SRC(insn->code) == BPF_X) {
+ if (BPF_CLASS(insn->code) == BPF_ALU64) {
+ /* case: R1 = R2
+ * copy register state to dest reg
+ */
+ regs[insn->dst_reg] = regs[insn->src_reg];
+ } else {
+ regs[insn->dst_reg].type = UNKNOWN_VALUE;
+ regs[insn->dst_reg].map_ptr = NULL;
+ }
+ } else {
+ /* case: R = imm
+ * remember the value we stored into this reg
+ */
+ regs[insn->dst_reg].type = CONST_IMM;
+ regs[insn->dst_reg].imm = insn->imm;
+ }
+
+ } else if (opcode > BPF_END) {
+ verbose("invalid BPF_ALU opcode %x\n", opcode);
+ return -EINVAL;
+
+ } else { /* all other ALU ops: and, sub, xor, add, ... */
+
+ bool stack_relative = false;
+
+ if (BPF_SRC(insn->code) == BPF_X) {
+ if (insn->imm != 0 || insn->off != 0) {
+ verbose("BPF_ALU uses reserved fields\n");
+ return -EINVAL;
+ }
+ /* check src1 operand */
+ _(check_reg_arg(regs, insn->src_reg, SRC_OP));
+ } else {
+ if (insn->src_reg != BPF_REG_0 || insn->off != 0) {
+ verbose("BPF_ALU uses reserved fields\n");
+ return -EINVAL;
+ }
+ }
+
+ /* check src2 operand */
+ _(check_reg_arg(regs, insn->dst_reg, SRC_OP));
+
+ if ((opcode == BPF_MOD || opcode == BPF_DIV) &&
+ BPF_SRC(insn->code) == BPF_K && insn->imm == 0) {
+ verbose("div by zero\n");
+ return -EINVAL;
+ }
+
+ /* pattern match 'bpf_add Rx, imm' instruction */
+ if (opcode == BPF_ADD && BPF_CLASS(insn->code) == BPF_ALU64 &&
+ regs[insn->dst_reg].type == FRAME_PTR &&
+ BPF_SRC(insn->code) == BPF_K)
+ stack_relative = true;
+
+ /* check dest operand */
+ _(check_reg_arg(regs, insn->dst_reg, DST_OP));
+
+ if (stack_relative) {
+ regs[insn->dst_reg].type = PTR_TO_STACK;
+ regs[insn->dst_reg].imm = insn->imm;
+ }
+ }
+
+ return 0;
+}
+
+static int check_cond_jmp_op(struct verifier_env *env,
+ struct bpf_insn *insn, int *insn_idx)
+{
+ struct reg_state *regs = env->cur_state.regs;
+ struct verifier_state *other_branch;
+ u8 opcode = BPF_OP(insn->code);
+
+ if (opcode > BPF_EXIT) {
+ verbose("invalid BPF_JMP opcode %x\n", opcode);
+ return -EINVAL;
+ }
+
+ if (BPF_SRC(insn->code) == BPF_X) {
+ if (insn->imm != 0) {
+ verbose("BPF_JMP uses reserved fields\n");
+ return -EINVAL;
+ }
+
+ /* check src1 operand */
+ _(check_reg_arg(regs, insn->src_reg, SRC_OP));
+ } else {
+ if (insn->src_reg != BPF_REG_0) {
+ verbose("BPF_JMP uses reserved fields\n");
+ return -EINVAL;
+ }
+ }
+
+ /* check src2 operand */
+ _(check_reg_arg(regs, insn->dst_reg, SRC_OP));
+
+ /* detect if R == 0 where R was initialized to zero earlier */
+ if (BPF_SRC(insn->code) == BPF_K &&
+ (opcode == BPF_JEQ || opcode == BPF_JNE) &&
+ regs[insn->dst_reg].type == CONST_IMM &&
+ regs[insn->dst_reg].imm == insn->imm) {
+ if (opcode == BPF_JEQ) {
+ /* if (imm == imm) goto pc+off;
+ * only follow the goto, ignore fall-through
+ */
+ *insn_idx += insn->off;
+ return 0;
+ } else {
+ /* if (imm != imm) goto pc+off;
+ * only follow fall-through branch, since
+ * that's where the program will go
+ */
+ return 0;
+ }
+ }
+
+ other_branch = push_stack(env, *insn_idx + insn->off + 1, *insn_idx);
+ if (!other_branch)
+ return -EFAULT;
+
+ /* detect if R == 0 where R is returned value from bpf_map_lookup_elem() */
+ if (BPF_SRC(insn->code) == BPF_K &&
+ insn->imm == 0 && (opcode == BPF_JEQ ||
+ opcode == BPF_JNE) &&
+ regs[insn->dst_reg].type == PTR_TO_MAP_VALUE_OR_NULL) {
+ if (opcode == BPF_JEQ) {
+ /* next fallthrough insn can access memory via
+ * this register
+ */
+ regs[insn->dst_reg].type = PTR_TO_MAP_VALUE;
+ /* branch targer cannot access it, since reg == 0 */
+ other_branch->regs[insn->dst_reg].type = CONST_IMM;
+ other_branch->regs[insn->dst_reg].imm = 0;
+ } else {
+ other_branch->regs[insn->dst_reg].type = PTR_TO_MAP_VALUE;
+ regs[insn->dst_reg].type = CONST_IMM;
+ regs[insn->dst_reg].imm = 0;
+ }
+ } else if (BPF_SRC(insn->code) == BPF_K &&
+ (opcode == BPF_JEQ || opcode == BPF_JNE)) {
+
+ if (opcode == BPF_JEQ) {
+ /* detect if (R == imm) goto
+ * and in the target state recognize that R = imm
+ */
+ other_branch->regs[insn->dst_reg].type = CONST_IMM;
+ other_branch->regs[insn->dst_reg].imm = insn->imm;
+ } else {
+ /* detect if (R != imm) goto
+ * and in the fall-through state recognize that R = imm
+ */
+ regs[insn->dst_reg].type = CONST_IMM;
+ regs[insn->dst_reg].imm = insn->imm;
+ }
+ }
+ if (log_level)
+ print_verifier_state(env);
+ return 0;
+}
+
/* return the map pointer stored inside BPF_LD_IMM64 instruction */
static struct bpf_map *ld_imm64_to_map_ptr(struct bpf_insn *insn)
{
@@ -313,6 +1110,34 @@ static struct bpf_map *ld_imm64_to_map_ptr(struct bpf_insn *insn)
return (struct bpf_map *) (unsigned long) imm64;
}
+/* verify BPF_LD_IMM64 instruction */
+static int check_ld_imm(struct verifier_env *env, struct bpf_insn *insn)
+{
+ struct reg_state *regs = env->cur_state.regs;
+
+ if (BPF_SIZE(insn->code) != BPF_DW) {
+ verbose("invalid BPF_LD_IMM insn\n");
+ return -EINVAL;
+ }
+ if (insn->off != 0) {
+ verbose("BPF_LD_IMM64 uses reserved fields\n");
+ return -EINVAL;
+ }
+
+ _(check_reg_arg(regs, insn->dst_reg, DST_OP));
+
+ if (insn->src_reg == 0)
+ /* generic move 64-bit immediate into a register */
+ return 0;
+
+ /* replace_map_fd_with_map_ptr() should have caught bad ld_imm64 */
+ BUG_ON(insn->src_reg != BPF_PSEUDO_MAP_FD);
+
+ regs[insn->dst_reg].type = CONST_PTR_TO_MAP;
+ regs[insn->dst_reg].map_ptr = ld_imm64_to_map_ptr(insn);
+ return 0;
+}
+
/* non-recursive DFS pseudo code
* 1 procedure DFS-iterative(G,v):
* 2 label v as discovered
@@ -492,6 +1317,181 @@ free_st:
return ret;
}
+static int do_check(struct verifier_env *env)
+{
+ struct verifier_state *state = &env->cur_state;
+ struct bpf_insn *insns = env->prog->insnsi;
+ struct reg_state *regs = state->regs;
+ int insn_cnt = env->prog->len;
+ int insn_idx, prev_insn_idx = 0;
+ int insn_processed = 0;
+ bool do_print_state = false;
+
+ init_reg_state(regs);
+ insn_idx = 0;
+ for (;;) {
+ struct bpf_insn *insn;
+ u8 class;
+
+ if (insn_idx >= insn_cnt) {
+ verbose("invalid insn idx %d insn_cnt %d\n",
+ insn_idx, insn_cnt);
+ return -EFAULT;
+ }
+
+ insn = &insns[insn_idx];
+ class = BPF_CLASS(insn->code);
+
+ if (++insn_processed > 32768) {
+ verbose("BPF program is too large. Proccessed %d insn\n",
+ insn_processed);
+ return -E2BIG;
+ }
+
+ if (log_level && do_print_state) {
+ verbose("\nfrom %d to %d:", prev_insn_idx, insn_idx);
+ print_verifier_state(env);
+ do_print_state = false;
+ }
+
+ if (log_level) {
+ verbose("%d: ", insn_idx);
+ print_bpf_insn(insn);
+ }
+
+ if (class == BPF_ALU || class == BPF_ALU64) {
+ _(check_alu_op(regs, insn));
+
+ } else if (class == BPF_LDX) {
+ if (BPF_MODE(insn->code) != BPF_MEM ||
+ insn->imm != 0) {
+ verbose("BPF_LDX uses reserved fields\n");
+ return -EINVAL;
+ }
+ /* check src operand */
+ _(check_reg_arg(regs, insn->src_reg, SRC_OP));
+
+ _(check_reg_arg(regs, insn->dst_reg, DST_OP_NO_MARK));
+
+ /* check that memory (src_reg + off) is readable,
+ * the state of dst_reg will be updated by this func
+ */
+ _(check_mem_access(env, insn->src_reg, insn->off,
+ BPF_SIZE(insn->code), BPF_READ,
+ insn->dst_reg));
+
+ } else if (class == BPF_STX) {
+ if (BPF_MODE(insn->code) == BPF_XADD) {
+ _(check_xadd(env, insn));
+ insn_idx++;
+ continue;
+ }
+
+ if (BPF_MODE(insn->code) != BPF_MEM ||
+ insn->imm != 0) {
+ verbose("BPF_STX uses reserved fields\n");
+ return -EINVAL;
+ }
+ /* check src1 operand */
+ _(check_reg_arg(regs, insn->src_reg, SRC_OP));
+ /* check src2 operand */
+ _(check_reg_arg(regs, insn->dst_reg, SRC_OP));
+
+ /* check that memory (dst_reg + off) is writeable */
+ _(check_mem_access(env, insn->dst_reg, insn->off,
+ BPF_SIZE(insn->code), BPF_WRITE,
+ insn->src_reg));
+
+ } else if (class == BPF_ST) {
+ if (BPF_MODE(insn->code) != BPF_MEM ||
+ insn->src_reg != BPF_REG_0) {
+ verbose("BPF_ST uses reserved fields\n");
+ return -EINVAL;
+ }
+ /* check src operand */
+ _(check_reg_arg(regs, insn->dst_reg, SRC_OP));
+
+ /* check that memory (dst_reg + off) is writeable */
+ _(check_mem_access(env, insn->dst_reg, insn->off,
+ BPF_SIZE(insn->code), BPF_WRITE,
+ -1));
+
+ } else if (class == BPF_JMP) {
+ u8 opcode = BPF_OP(insn->code);
+
+ if (opcode == BPF_CALL) {
+ if (BPF_SRC(insn->code) != BPF_K ||
+ insn->off != 0 ||
+ insn->src_reg != BPF_REG_0 ||
+ insn->dst_reg != BPF_REG_0) {
+ verbose("BPF_CALL uses reserved fields\n");
+ return -EINVAL;
+ }
+
+ _(check_call(env, insn->imm));
+
+ } else if (opcode == BPF_JA) {
+ if (BPF_SRC(insn->code) != BPF_K ||
+ insn->imm != 0 ||
+ insn->src_reg != BPF_REG_0 ||
+ insn->dst_reg != BPF_REG_0) {
+ verbose("BPF_JA uses reserved fields\n");
+ return -EINVAL;
+ }
+
+ insn_idx += insn->off + 1;
+ continue;
+
+ } else if (opcode == BPF_EXIT) {
+ if (BPF_SRC(insn->code) != BPF_K ||
+ insn->imm != 0 ||
+ insn->src_reg != BPF_REG_0 ||
+ insn->dst_reg != BPF_REG_0) {
+ verbose("BPF_EXIT uses reserved fields\n");
+ return -EINVAL;
+ }
+
+ /* eBPF calling convetion is such that R0 is used
+ * to return the value from eBPF program.
+ * Make sure that it's readable at this time
+ * of bpf_exit, which means that program wrote
+ * something into it earlier
+ */
+ _(check_reg_arg(regs, BPF_REG_0, SRC_OP));
+ insn_idx = pop_stack(env, &prev_insn_idx);
+ if (insn_idx < 0) {
+ break;
+ } else {
+ do_print_state = true;
+ continue;
+ }
+ } else {
+ _(check_cond_jmp_op(env, insn, &insn_idx));
+ }
+ } else if (class == BPF_LD) {
+ u8 mode = BPF_MODE(insn->code);
+
+ if (mode == BPF_ABS || mode == BPF_IND) {
+ verbose("LD_ABS is not supported yet\n");
+ return -EINVAL;
+ } else if (mode == BPF_IMM) {
+ _(check_ld_imm(env, insn));
+ insn_idx++;
+ } else {
+ verbose("invalid BPF_LD mode\n");
+ return -EINVAL;
+ }
+ } else {
+ verbose("unknown insn class %d\n", class);
+ return -EINVAL;
+ }
+
+ insn_idx++;
+ }
+
+ return 0;
+}
+
/* look for pseudo eBPF instructions that access map FDs and
* replace them with actual map pointers
*/
@@ -645,9 +1645,10 @@ int bpf_check(struct bpf_prog *prog, union bpf_attr *attr)
if (ret < 0)
goto skip_full_check;
- /* ret = do_check(env); */
+ ret = do_check(env);
skip_full_check:
+ while (pop_stack(env, NULL) >= 0);
if (log_level && log_len >= log_size - 1) {
BUG_ON(log_len >= log_size);
--
1.7.9.5
^ permalink raw reply related
* [PATCH v14 net-next 11/11] bpf: mini eBPF library, test stubs and verifier testsuite
From: Alexei Starovoitov @ 2014-09-22 0:06 UTC (permalink / raw)
To: David S. Miller
Cc: Ingo Molnar, Linus Torvalds, Andy Lutomirski, Daniel Borkmann,
Hannes Frederic Sowa, Chema Gonzalez, Eric Dumazet,
Peter Zijlstra, Pablo Neira Ayuso, H. Peter Anvin, Andrew Morton,
Kees Cook, linux-api, netdev, linux-kernel
In-Reply-To: <1411344411-3824-1-git-send-email-ast@plumgrid.com>
1.
the library includes a trivial set of BPF syscall wrappers:
int bpf_create_map(int key_size, int value_size, int max_entries);
int bpf_update_elem(int fd, void *key, void *value);
int bpf_lookup_elem(int fd, void *key, void *value);
int bpf_delete_elem(int fd, void *key);
int bpf_get_next_key(int fd, void *key, void *next_key);
int bpf_prog_load(enum bpf_prog_type prog_type,
const struct sock_filter_int *insns, int insn_len,
const char *license);
bpf_prog_load() stores verifier log into global bpf_log_buf[] array
and BPF_*() macros to build instructions
2.
test stubs configure eBPF infra with 'unspec' map and program types.
These are fake types used by user space testsuite only.
3.
verifier tests valid and invalid programs and expects predefined
error log messages from kernel.
40 tests so far.
$ sudo ./test_verifier
#0 add+sub+mul OK
#1 unreachable OK
#2 unreachable2 OK
#3 out of range jump OK
#4 out of range jump2 OK
#5 test1 ld_imm64 OK
...
Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
---
kernel/bpf/Makefile | 4 +
kernel/bpf/test_stub.c | 116 +++++++++
lib/Kconfig.debug | 3 +-
samples/bpf/Makefile | 12 +
samples/bpf/libbpf.c | 94 ++++++++
samples/bpf/libbpf.h | 172 ++++++++++++++
samples/bpf/test_verifier.c | 548 +++++++++++++++++++++++++++++++++++++++++++
7 files changed, 948 insertions(+), 1 deletion(-)
create mode 100644 kernel/bpf/test_stub.c
create mode 100644 samples/bpf/Makefile
create mode 100644 samples/bpf/libbpf.c
create mode 100644 samples/bpf/libbpf.h
create mode 100644 samples/bpf/test_verifier.c
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 3c726b0995b7..45427239f375 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -1 +1,5 @@
obj-y := core.o syscall.o verifier.o
+
+ifdef CONFIG_TEST_BPF
+obj-y += test_stub.o
+endif
diff --git a/kernel/bpf/test_stub.c b/kernel/bpf/test_stub.c
new file mode 100644
index 000000000000..fcaddff4003e
--- /dev/null
+++ b/kernel/bpf/test_stub.c
@@ -0,0 +1,116 @@
+/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/err.h>
+#include <linux/bpf.h>
+
+/* test stubs for BPF_MAP_TYPE_UNSPEC and for BPF_PROG_TYPE_UNSPEC
+ * to be used by user space verifier testsuite
+ */
+struct bpf_context {
+ u64 arg1;
+ u64 arg2;
+};
+
+static u64 test_func(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5)
+{
+ return 0;
+}
+
+static struct bpf_func_proto test_funcs[] = {
+ [BPF_FUNC_unspec] = {
+ .func = test_func,
+ .gpl_only = true,
+ .ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
+ .arg1_type = ARG_CONST_MAP_PTR,
+ .arg2_type = ARG_PTR_TO_MAP_KEY,
+ },
+};
+
+static const struct bpf_func_proto *test_func_proto(enum bpf_func_id func_id)
+{
+ if (func_id < 0 || func_id >= ARRAY_SIZE(test_funcs))
+ return NULL;
+ return &test_funcs[func_id];
+}
+
+static const struct bpf_context_access {
+ int size;
+ enum bpf_access_type type;
+} test_ctx_access[] = {
+ [offsetof(struct bpf_context, arg1)] = {
+ FIELD_SIZEOF(struct bpf_context, arg1),
+ BPF_READ
+ },
+ [offsetof(struct bpf_context, arg2)] = {
+ FIELD_SIZEOF(struct bpf_context, arg2),
+ BPF_READ
+ },
+};
+
+static bool test_is_valid_access(int off, int size, enum bpf_access_type type)
+{
+ const struct bpf_context_access *access;
+
+ if (off < 0 || off >= ARRAY_SIZE(test_ctx_access))
+ return false;
+
+ access = &test_ctx_access[off];
+ if (access->size == size && (access->type & type))
+ return true;
+
+ return false;
+}
+
+static struct bpf_verifier_ops test_ops = {
+ .get_func_proto = test_func_proto,
+ .is_valid_access = test_is_valid_access,
+};
+
+static struct bpf_prog_type_list tl_prog = {
+ .ops = &test_ops,
+ .type = BPF_PROG_TYPE_UNSPEC,
+};
+
+static struct bpf_map *test_map_alloc(union bpf_attr *attr)
+{
+ struct bpf_map *map;
+
+ map = kzalloc(sizeof(*map), GFP_USER);
+ if (!map)
+ return ERR_PTR(-ENOMEM);
+
+ map->key_size = attr->key_size;
+ map->value_size = attr->value_size;
+ map->max_entries = attr->max_entries;
+ return map;
+}
+
+static void test_map_free(struct bpf_map *map)
+{
+ kfree(map);
+}
+
+static struct bpf_map_ops test_map_ops = {
+ .map_alloc = test_map_alloc,
+ .map_free = test_map_free,
+};
+
+static struct bpf_map_type_list tl_map = {
+ .ops = &test_map_ops,
+ .type = BPF_MAP_TYPE_UNSPEC,
+};
+
+static int __init register_test_ops(void)
+{
+ bpf_register_map_type(&tl_map);
+ bpf_register_prog_type(&tl_prog);
+ return 0;
+}
+late_initcall(register_test_ops);
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index a28590083622..3ac43f34437b 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1672,7 +1672,8 @@ config TEST_BPF
against the BPF interpreter or BPF JIT compiler depending on the
current setting. This is in particular useful for BPF JIT compiler
development, but also to run regression tests against changes in
- the interpreter code.
+ the interpreter code. It also enables test stubs for eBPF maps and
+ verifier used by user space verifier testsuite.
If unsure, say N.
diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
new file mode 100644
index 000000000000..634391797856
--- /dev/null
+++ b/samples/bpf/Makefile
@@ -0,0 +1,12 @@
+# kbuild trick to avoid linker error. Can be omitted if a module is built.
+obj- := dummy.o
+
+# List of programs to build
+hostprogs-y := test_verifier
+
+test_verifier-objs := test_verifier.o libbpf.o
+
+# Tell kbuild to always build the programs
+always := $(hostprogs-y)
+
+HOSTCFLAGS += -I$(objtree)/usr/include
diff --git a/samples/bpf/libbpf.c b/samples/bpf/libbpf.c
new file mode 100644
index 000000000000..ff6504420738
--- /dev/null
+++ b/samples/bpf/libbpf.c
@@ -0,0 +1,94 @@
+/* eBPF mini library */
+#include <stdlib.h>
+#include <stdio.h>
+#include <linux/unistd.h>
+#include <unistd.h>
+#include <string.h>
+#include <linux/netlink.h>
+#include <linux/bpf.h>
+#include <errno.h>
+#include "libbpf.h"
+
+static __u64 ptr_to_u64(void *ptr)
+{
+ return (__u64) (unsigned long) ptr;
+}
+
+int bpf_create_map(enum bpf_map_type map_type, int key_size, int value_size,
+ int max_entries)
+{
+ union bpf_attr attr = {
+ .map_type = map_type,
+ .key_size = key_size,
+ .value_size = value_size,
+ .max_entries = max_entries
+ };
+
+ return syscall(__NR_bpf, BPF_MAP_CREATE, &attr, sizeof(attr));
+}
+
+int bpf_update_elem(int fd, void *key, void *value)
+{
+ union bpf_attr attr = {
+ .map_fd = fd,
+ .key = ptr_to_u64(key),
+ .value = ptr_to_u64(value),
+ };
+
+ return syscall(__NR_bpf, BPF_MAP_UPDATE_ELEM, &attr, sizeof(attr));
+}
+
+int bpf_lookup_elem(int fd, void *key, void *value)
+{
+ union bpf_attr attr = {
+ .map_fd = fd,
+ .key = ptr_to_u64(key),
+ .value = ptr_to_u64(value),
+ };
+
+ return syscall(__NR_bpf, BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
+}
+
+int bpf_delete_elem(int fd, void *key)
+{
+ union bpf_attr attr = {
+ .map_fd = fd,
+ .key = ptr_to_u64(key),
+ };
+
+ return syscall(__NR_bpf, BPF_MAP_DELETE_ELEM, &attr, sizeof(attr));
+}
+
+int bpf_get_next_key(int fd, void *key, void *next_key)
+{
+ union bpf_attr attr = {
+ .map_fd = fd,
+ .key = ptr_to_u64(key),
+ .next_key = ptr_to_u64(next_key),
+ };
+
+ return syscall(__NR_bpf, BPF_MAP_GET_NEXT_KEY, &attr, sizeof(attr));
+}
+
+#define ROUND_UP(x, n) (((x) + (n) - 1u) & ~((n) - 1u))
+
+char bpf_log_buf[LOG_BUF_SIZE];
+
+int bpf_prog_load(enum bpf_prog_type prog_type,
+ const struct bpf_insn *insns, int prog_len,
+ const char *license)
+{
+ union bpf_attr attr = {
+ .prog_type = prog_type,
+ .insns = ptr_to_u64((void *) insns),
+ .insn_cnt = prog_len / sizeof(struct bpf_insn),
+ .license = ptr_to_u64((void *) license),
+ .log_buf = ptr_to_u64(bpf_log_buf),
+ .log_size = LOG_BUF_SIZE,
+ .log_level = 1,
+ };
+
+ bpf_log_buf[0] = 0;
+
+ return syscall(__NR_bpf, BPF_PROG_LOAD, &attr, sizeof(attr));
+}
diff --git a/samples/bpf/libbpf.h b/samples/bpf/libbpf.h
new file mode 100644
index 000000000000..8a31babeca5d
--- /dev/null
+++ b/samples/bpf/libbpf.h
@@ -0,0 +1,172 @@
+/* eBPF mini library */
+#ifndef __LIBBPF_H
+#define __LIBBPF_H
+
+struct bpf_insn;
+
+int bpf_create_map(enum bpf_map_type map_type, int key_size, int value_size,
+ int max_entries);
+int bpf_update_elem(int fd, void *key, void *value);
+int bpf_lookup_elem(int fd, void *key, void *value);
+int bpf_delete_elem(int fd, void *key);
+int bpf_get_next_key(int fd, void *key, void *next_key);
+
+int bpf_prog_load(enum bpf_prog_type prog_type,
+ const struct bpf_insn *insns, int insn_len,
+ const char *license);
+
+#define LOG_BUF_SIZE 8192
+extern char bpf_log_buf[LOG_BUF_SIZE];
+
+/* ALU ops on registers, bpf_add|sub|...: dst_reg += src_reg */
+
+#define BPF_ALU64_REG(OP, DST, SRC) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU64 | BPF_OP(OP) | BPF_X, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = 0 })
+
+#define BPF_ALU32_REG(OP, DST, SRC) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU | BPF_OP(OP) | BPF_X, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = 0 })
+
+/* ALU ops on immediates, bpf_add|sub|...: dst_reg += imm32 */
+
+#define BPF_ALU64_IMM(OP, DST, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU64 | BPF_OP(OP) | BPF_K, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = IMM })
+
+#define BPF_ALU32_IMM(OP, DST, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU | BPF_OP(OP) | BPF_K, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = IMM })
+
+/* Short form of mov, dst_reg = src_reg */
+
+#define BPF_MOV64_REG(DST, SRC) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU64 | BPF_MOV | BPF_X, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = 0 })
+
+/* Short form of mov, dst_reg = imm32 */
+
+#define BPF_MOV64_IMM(DST, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU64 | BPF_MOV | BPF_K, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = IMM })
+
+/* BPF_LD_IMM64 macro encodes single 'load 64-bit immediate' insn */
+#define BPF_LD_IMM64(DST, IMM) \
+ BPF_LD_IMM64_RAW(DST, 0, IMM)
+
+#define BPF_LD_IMM64_RAW(DST, SRC, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_LD | BPF_DW | BPF_IMM, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = (__u32) (IMM) }), \
+ ((struct bpf_insn) { \
+ .code = 0, /* zero is reserved opcode */ \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = ((__u64) (IMM)) >> 32 })
+
+#define BPF_PSEUDO_MAP_FD 1
+
+/* pseudo BPF_LD_IMM64 insn used to refer to process-local map_fd */
+#define BPF_LD_MAP_FD(DST, MAP_FD) \
+ BPF_LD_IMM64_RAW(DST, BPF_PSEUDO_MAP_FD, MAP_FD)
+
+
+/* Memory load, dst_reg = *(uint *) (src_reg + off16) */
+
+#define BPF_LDX_MEM(SIZE, DST, SRC, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_LDX | BPF_SIZE(SIZE) | BPF_MEM, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = 0 })
+
+/* Memory store, *(uint *) (dst_reg + off16) = src_reg */
+
+#define BPF_STX_MEM(SIZE, DST, SRC, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_STX | BPF_SIZE(SIZE) | BPF_MEM, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = 0 })
+
+/* Memory store, *(uint *) (dst_reg + off16) = imm32 */
+
+#define BPF_ST_MEM(SIZE, DST, OFF, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ST | BPF_SIZE(SIZE) | BPF_MEM, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = OFF, \
+ .imm = IMM })
+
+/* Conditional jumps against registers, if (dst_reg 'op' src_reg) goto pc + off16 */
+
+#define BPF_JMP_REG(OP, DST, SRC, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_OP(OP) | BPF_X, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = 0 })
+
+/* Conditional jumps against immediates, if (dst_reg 'op' imm32) goto pc + off16 */
+
+#define BPF_JMP_IMM(OP, DST, IMM, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_OP(OP) | BPF_K, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = OFF, \
+ .imm = IMM })
+
+/* Raw code statement block */
+
+#define BPF_RAW_INSN(CODE, DST, SRC, OFF, IMM) \
+ ((struct bpf_insn) { \
+ .code = CODE, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = IMM })
+
+/* Program exit */
+
+#define BPF_EXIT_INSN() \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_EXIT, \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = 0 })
+
+#endif
diff --git a/samples/bpf/test_verifier.c b/samples/bpf/test_verifier.c
new file mode 100644
index 000000000000..d10992e2740e
--- /dev/null
+++ b/samples/bpf/test_verifier.c
@@ -0,0 +1,548 @@
+/*
+ * Testsuite for eBPF verifier
+ *
+ * Copyright (c) 2014 PLUMgrid, http://plumgrid.com
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <stdio.h>
+#include <unistd.h>
+#include <linux/bpf.h>
+#include <errno.h>
+#include <linux/unistd.h>
+#include <string.h>
+#include <linux/filter.h>
+#include "libbpf.h"
+
+#define MAX_INSNS 512
+#define ARRAY_SIZE(x) (sizeof(x) / sizeof(*(x)))
+
+struct bpf_test {
+ const char *descr;
+ struct bpf_insn insns[MAX_INSNS];
+ int fixup[32];
+ const char *errstr;
+ enum {
+ ACCEPT,
+ REJECT
+ } result;
+};
+
+static struct bpf_test tests[] = {
+ {
+ "add+sub+mul",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_1, 1),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 2),
+ BPF_MOV64_IMM(BPF_REG_2, 3),
+ BPF_ALU64_REG(BPF_SUB, BPF_REG_1, BPF_REG_2),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -1),
+ BPF_ALU64_IMM(BPF_MUL, BPF_REG_1, 3),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_1),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ },
+ {
+ "unreachable",
+ .insns = {
+ BPF_EXIT_INSN(),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "unreachable",
+ .result = REJECT,
+ },
+ {
+ "unreachable2",
+ .insns = {
+ BPF_JMP_IMM(BPF_JA, 0, 0, 1),
+ BPF_JMP_IMM(BPF_JA, 0, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "unreachable",
+ .result = REJECT,
+ },
+ {
+ "out of range jump",
+ .insns = {
+ BPF_JMP_IMM(BPF_JA, 0, 0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "jump out of range",
+ .result = REJECT,
+ },
+ {
+ "out of range jump2",
+ .insns = {
+ BPF_JMP_IMM(BPF_JA, 0, 0, -2),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "jump out of range",
+ .result = REJECT,
+ },
+ {
+ "test1 ld_imm64",
+ .insns = {
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1),
+ BPF_LD_IMM64(BPF_REG_0, 0),
+ BPF_LD_IMM64(BPF_REG_0, 0),
+ BPF_LD_IMM64(BPF_REG_0, 1),
+ BPF_LD_IMM64(BPF_REG_0, 1),
+ BPF_MOV64_IMM(BPF_REG_0, 2),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid BPF_LD_IMM insn",
+ .result = REJECT,
+ },
+ {
+ "test2 ld_imm64",
+ .insns = {
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1),
+ BPF_LD_IMM64(BPF_REG_0, 0),
+ BPF_LD_IMM64(BPF_REG_0, 0),
+ BPF_LD_IMM64(BPF_REG_0, 1),
+ BPF_LD_IMM64(BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid BPF_LD_IMM insn",
+ .result = REJECT,
+ },
+ {
+ "test3 ld_imm64",
+ .insns = {
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1),
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, 0, 0, 0),
+ BPF_LD_IMM64(BPF_REG_0, 0),
+ BPF_LD_IMM64(BPF_REG_0, 0),
+ BPF_LD_IMM64(BPF_REG_0, 1),
+ BPF_LD_IMM64(BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_ld_imm64 insn",
+ .result = REJECT,
+ },
+ {
+ "test4 ld_imm64",
+ .insns = {
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, 0, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_ld_imm64 insn",
+ .result = REJECT,
+ },
+ {
+ "test5 ld_imm64",
+ .insns = {
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, 0, 0, 0),
+ },
+ .errstr = "invalid bpf_ld_imm64 insn",
+ .result = REJECT,
+ },
+ {
+ "no bpf_exit",
+ .insns = {
+ BPF_ALU64_REG(BPF_MOV, BPF_REG_0, BPF_REG_2),
+ },
+ .errstr = "jump out of range",
+ .result = REJECT,
+ },
+ {
+ "loop (back-edge)",
+ .insns = {
+ BPF_JMP_IMM(BPF_JA, 0, 0, -1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "back-edge",
+ .result = REJECT,
+ },
+ {
+ "loop2 (back-edge)",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_3, BPF_REG_0),
+ BPF_JMP_IMM(BPF_JA, 0, 0, -4),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "back-edge",
+ .result = REJECT,
+ },
+ {
+ "conditional loop",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_3, BPF_REG_0),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, -3),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "back-edge",
+ .result = REJECT,
+ },
+ {
+ "read uninitialized register",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_2),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R2 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "read invalid register",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_0, -1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R15 is invalid",
+ .result = REJECT,
+ },
+ {
+ "program doesn't init R0 before exit",
+ .insns = {
+ BPF_ALU64_REG(BPF_MOV, BPF_REG_2, BPF_REG_1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R0 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "stack out of bounds",
+ .insns = {
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, 8, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid stack",
+ .result = REJECT,
+ },
+ {
+ "invalid call insn1",
+ .insns = {
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL | BPF_X, 0, 0, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "BPF_CALL uses reserved",
+ .result = REJECT,
+ },
+ {
+ "invalid call insn2",
+ .insns = {
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 1, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "BPF_CALL uses reserved",
+ .result = REJECT,
+ },
+ {
+ "invalid function call",
+ .insns = {
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, 1234567),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid func 1234567",
+ .result = REJECT,
+ },
+ {
+ "uninitialized stack1",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_unspec),
+ BPF_EXIT_INSN(),
+ },
+ .fixup = {2},
+ .errstr = "invalid indirect read from stack",
+ .result = REJECT,
+ },
+ {
+ "uninitialized stack2",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, -8),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid read from stack",
+ .result = REJECT,
+ },
+ {
+ "check valid spill/fill",
+ .insns = {
+ /* spill R1(ctx) into stack */
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_1, -8),
+
+ /* fill it back into R2 */
+ BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_10, -8),
+
+ /* should be able to access R0 = *(R2 + 8) */
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 8),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ },
+ {
+ "check corrupted spill/fill",
+ .insns = {
+ /* spill R1(ctx) into stack */
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_1, -8),
+
+ /* mess up with R1 pointer on stack */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -7, 0x23),
+
+ /* fill back into R0 should fail */
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_10, -8),
+
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "corrupted spill",
+ .result = REJECT,
+ },
+ {
+ "invalid src register in STX",
+ .insns = {
+ BPF_STX_MEM(BPF_B, BPF_REG_10, -1, -1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R15 is invalid",
+ .result = REJECT,
+ },
+ {
+ "invalid dst register in STX",
+ .insns = {
+ BPF_STX_MEM(BPF_B, 14, BPF_REG_10, -1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R14 is invalid",
+ .result = REJECT,
+ },
+ {
+ "invalid dst register in ST",
+ .insns = {
+ BPF_ST_MEM(BPF_B, 14, -1, -1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R14 is invalid",
+ .result = REJECT,
+ },
+ {
+ "invalid src register in LDX",
+ .insns = {
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, 12, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R12 is invalid",
+ .result = REJECT,
+ },
+ {
+ "invalid dst register in LDX",
+ .insns = {
+ BPF_LDX_MEM(BPF_B, 11, BPF_REG_1, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R11 is invalid",
+ .result = REJECT,
+ },
+ {
+ "junk insn",
+ .insns = {
+ BPF_RAW_INSN(0, 0, 0, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid BPF_LD_IMM",
+ .result = REJECT,
+ },
+ {
+ "junk insn2",
+ .insns = {
+ BPF_RAW_INSN(1, 0, 0, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "BPF_LDX uses reserved fields",
+ .result = REJECT,
+ },
+ {
+ "junk insn3",
+ .insns = {
+ BPF_RAW_INSN(-1, 0, 0, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid BPF_ALU opcode f0",
+ .result = REJECT,
+ },
+ {
+ "junk insn4",
+ .insns = {
+ BPF_RAW_INSN(-1, -1, -1, -1, -1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid BPF_ALU opcode f0",
+ .result = REJECT,
+ },
+ {
+ "junk insn5",
+ .insns = {
+ BPF_RAW_INSN(0x7f, -1, -1, -1, -1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "BPF_ALU uses reserved fields",
+ .result = REJECT,
+ },
+ {
+ "misaligned read from stack",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, -4),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "misaligned access",
+ .result = REJECT,
+ },
+ {
+ "invalid map_fd for function call",
+ .insns = {
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_ALU64_REG(BPF_MOV, BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_unspec),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "fd 0 is not pointing to valid bpf_map",
+ .result = REJECT,
+ },
+ {
+ "don't check return value before access",
+ .insns = {
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_unspec),
+ BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup = {3},
+ .errstr = "R0 invalid mem access 'map_value_or_null'",
+ .result = REJECT,
+ },
+ {
+ "access memory with incorrect alignment",
+ .insns = {
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_unspec),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
+ BPF_ST_MEM(BPF_DW, BPF_REG_0, 4, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup = {3},
+ .errstr = "misaligned access",
+ .result = REJECT,
+ },
+ {
+ "sometimes access memory with incorrect alignment",
+ .insns = {
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_unspec),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
+ BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0),
+ BPF_EXIT_INSN(),
+ BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .fixup = {3},
+ .errstr = "R0 invalid mem access",
+ .result = REJECT,
+ },
+};
+
+static int probe_filter_length(struct bpf_insn *fp)
+{
+ int len = 0;
+
+ for (len = MAX_INSNS - 1; len > 0; --len)
+ if (fp[len].code != 0 || fp[len].imm != 0)
+ break;
+
+ return len + 1;
+}
+
+static int create_map(void)
+{
+ long long key, value = 0;
+ int map_fd;
+
+ map_fd = bpf_create_map(BPF_MAP_TYPE_UNSPEC, sizeof(key), sizeof(value), 1024);
+ if (map_fd < 0) {
+ printf("failed to create map '%s'\n", strerror(errno));
+ }
+
+ return map_fd;
+}
+
+static int test(void)
+{
+ int prog_fd, i;
+
+ for (i = 0; i < ARRAY_SIZE(tests); i++) {
+ struct bpf_insn *prog = tests[i].insns;
+ int prog_len = probe_filter_length(prog);
+ int *fixup = tests[i].fixup;
+ int map_fd = -1;
+
+ if (*fixup) {
+ map_fd = create_map();
+
+ do {
+ prog[*fixup].imm = map_fd;
+ fixup++;
+ } while (*fixup);
+ }
+ printf("#%d %s ", i, tests[i].descr);
+
+ prog_fd = bpf_prog_load(BPF_PROG_TYPE_UNSPEC, prog,
+ prog_len * sizeof(struct bpf_insn),
+ "GPL");
+
+ if (tests[i].result == ACCEPT) {
+ if (prog_fd < 0) {
+ printf("FAIL\nfailed to load prog '%s'\n",
+ strerror(errno));
+ printf("%s", bpf_log_buf);
+ goto fail;
+ }
+ } else {
+ if (prog_fd >= 0) {
+ printf("FAIL\nunexpected success to load\n");
+ printf("%s", bpf_log_buf);
+ goto fail;
+ }
+ if (strstr(bpf_log_buf, tests[i].errstr) == 0) {
+ printf("FAIL\nunexpected error message: %s",
+ bpf_log_buf);
+ goto fail;
+ }
+ }
+
+ printf("OK\n");
+fail:
+ if (map_fd >= 0)
+ close(map_fd);
+ close(prog_fd);
+
+ }
+
+ return 0;
+}
+
+int main(void)
+{
+ return test();
+}
--
1.7.9.5
^ permalink raw reply related
* Re: [PATCH v3 1/5] drm/rockchip: Add basic drm driver
From: Mark yao @ 2014-09-22 1:27 UTC (permalink / raw)
To: David Herrmann
Cc: heiko-4mtYJXux2i+zQB+pC5nmwQ, Boris BREZILLON, David Airlie,
Rob Clark, Rob Herring, Pawel Moll, Mark Rutland, Ian Campbell,
Kumar Gala, Randy Dunlap, Grant Likely, Greg Kroah-Hartman,
John Stultz, Rom Lemarchand, devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-doc-u79uwXL29TY76Z2rM5mHXA, linux-kernel,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org,
Linux API, linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
dianders-F7+t8E8rja9g9hUCZPvPmw, Stéphane Marchesin,
dbehr-F7+t8E8rja9g9hUCZPvPmw
In-Reply-To: <CANq1E4Ti0yOQ+jCiXy7=NtFQ9cbWLOSLf6q-HWVdDFtMB1n29Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On 2014年09月19日 21:04, David Herrmann wrote:
> Hi
>
> On Fri, Sep 19, 2014 at 7:47 AM, Mark yao <mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org> wrote:
> [snip]
>> +static int rockchip_drm_bind(struct device *dev)
>> +{
>> + return drm_platform_init(&rockchip_drm_driver, to_platform_device(dev));
> Please avoid drm_platform_*() usage. We're about to drop all the
> drm_bus midlayers. See the tegra driver how to do it, but basically
> just this:
OK, I will do it.
> struct drm_device *ddev;
>
> ddev = drm_dev_alloc(&rockchip_drm_driver, &to_platform_device(dev)->dev);
> if (!ddev)
> return -ENOMEM;
>
> r = drm_dev_set_unique(ddev, dev_name(&ddev->dev));
> if (r < 0) {
> drm_dev_unref(ddev);
> return r;
> }
>
> r = drm_dev_register(ddev);
> if (r < 0) {
> drm_dev_unref(ddev);
> return r;
> }
>
>> +}
>> +
>> +static void rockchip_drm_unbind(struct device *dev)
>> +{
>> + drm_put_dev(dev_get_drvdata(dev));
> Please use:
>
> struct drm_device *ddev = dev_get_drvdata(dev);
>
> drm_dev_unregister(ddev);
> drm_dev_unref(ddev);
>
>
> Thanks
> David
>
>
>
^ permalink raw reply
* Re: [PATCH v3 1/5] drm/rockchip: Add basic drm driver
From: Mark yao @ 2014-09-22 1:32 UTC (permalink / raw)
To: Rob Clark
Cc: Heiko Stübner, Boris BREZILLON, David Airlie, Rob Herring,
Pawel Moll, Mark Rutland, Ian Campbell, Kumar Gala, Randy Dunlap,
Grant Likely, Greg Kroah-Hartman, John Stultz, Rom Lemarchand,
devicetree-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-doc-u79uwXL29TY76Z2rM5mHXA, Linux Kernel Mailing List,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
dianders-F7+t8E8rja9g9hUCZPvPmw, Stéphane Marchesin
In-Reply-To: <CAF6AEGsPoj6+PdKCJk5BPPzHS51Tz1FGqS2SjpartaRw7i=Pxg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On 2014年09月20日 08:03, Rob Clark wrote:
> On Fri, Sep 19, 2014 at 1:47 AM, Mark yao <mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org> wrote:
>> diff --git a/include/uapi/drm/rockchip_drm.h b/include/uapi/drm/rockchip_drm.h
>> new file mode 100644
>> index 0000000..8f8e60e
>> --- /dev/null
>> +++ b/include/uapi/drm/rockchip_drm.h
>> @@ -0,0 +1,97 @@
>> +/*
>> + *
>> + * Copyright (c) Fuzhou Rockchip Electronics Co.Ltd
>> + * Authors:
>> + * Mark Yao <yzq-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
>> + *
>> + * base on exynos_drm.h
>> + *
>> + * This program is free software; you can redistribute it and/or modify it
>> + * under the terms of the GNU General Public License as published by the
>> + * Free Software Foundation; either version 2 of the License, or (at your
>> + * option) any later version.
>> + */
>> +
>> +#ifndef _UAPI_ROCKCHIP_DRM_H
>> +#define _UAPI_ROCKCHIP_DRM_H
>> +
>> +#include <drm/drm.h>
>> +
>> +/**
>> + * User-desired buffer creation information structure.
>> + *
>> + * @size: user-desired memory allocation size.
>> + * @flags: user request for setting memory type or cache attributes.
>> + * @handle: returned a handle to created gem object.
>> + * - this handle will be set by gem module of kernel side.
>> + */
>> +struct drm_rockchip_gem_create {
>> + uint64_t size;
>> + uint32_t flags;
>> + uint32_t handle;
>> +};
>> +
>> +/**
>> + * A structure for getting buffer offset.
>> + *
>> + * @handle: a pointer to gem object created.
>> + * @pad: just padding to be 64-bit aligned.
>> + * @offset: relatived offset value of the memory region allocated.
>> + * - this value should be set by user.
>> + */
>> +struct drm_rockchip_gem_map_off {
>> + uint32_t handle;
>> + uint32_t pad;
>> + uint64_t offset;
>> +};
>> +
>> +/**
>> + * A structure for mapping buffer.
>> + *
>> + * @handle: a handle to gem object created.
>> + * @pad: just padding to be 64-bit aligned.
>> + * @size: memory size to be mapped.
>> + * @mapped: having user virtual address mmaped.
>> + * - this variable would be filled by rockchip gem module
>> + * of kernel side with user virtual address which is allocated
>> + * by do_mmap().
>> + */
>> +struct drm_rockchip_gem_mmap {
>> + uint32_t handle;
>> + uint32_t pad;
>> + uint64_t size;
>> + uint64_t mapped;
>> +};
> Could we do without the mmap ioctl? It has been a source of problems
> in other drivers, and the ioctl to get mmap offset, plus normal mmap()
> on drm device file should be sufficient
>
> BR,
> -R
OK, I will try to move the special mmap ioctl and use drm generic mmap
interface.
>
>> +/**
>> + * A structure to gem information.
>> + *
>> + * @handle: a handle to gem object created.
>> + * @flags: flag value including memory type and cache attribute and
>> + * this value would be set by driver.
>> + * @size: size to memory region allocated by gem and this size would
>> + * be set by driver.
>> + */
>> +struct drm_rockchip_gem_info {
>> + uint32_t handle;
>> + uint32_t flags;
>> + uint64_t size;
>> +};
>> +
>> +#define DRM_ROCKCHIP_GEM_CREATE 0x00
>> +#define DRM_ROCKCHIP_GEM_MAP_OFFSET 0x01
>> +#define DRM_ROCKCHIP_GEM_MMAP 0x02
>> +#define DRM_ROCKCHIP_GEM_GET 0x04
>> +
>> +#define DRM_IOCTL_ROCKCHIP_GEM_CREATE DRM_IOWR(DRM_COMMAND_BASE + \
>> + DRM_ROCKCHIP_GEM_CREATE, struct drm_rockchip_gem_create)
>> +
>> +#define DRM_IOCTL_ROCKCHIP_GEM_MAP_OFFSET DRM_IOWR(DRM_COMMAND_BASE + \
>> + DRM_ROCKCHIP_GEM_MAP_OFFSET, struct drm_rockchip_gem_map_off)
>> +
>> +#define DRM_IOCTL_ROCKCHIP_GEM_MMAP DRM_IOWR(DRM_COMMAND_BASE + \
>> + DRM_ROCKCHIP_GEM_MMAP, struct drm_rockchip_gem_mmap)
>> +
>> +#define DRM_IOCTL_ROCKCHIP_GEM_GET DRM_IOWR(DRM_COMMAND_BASE + \
>> + DRM_ROCKCHIP_GEM_GET, struct drm_rockchip_gem_info)
>> +#endif /* _UAPI_ROCKCHIP_DRM_H */
>
>
^ permalink raw reply
* Re: [PATCH v4 04/11] drivers: base: support cpu cache information interface to userspace via sysfs
From: Sudeep Holla @ 2014-09-22 8:55 UTC (permalink / raw)
To: Stephen Boyd, LKML
Cc: Sudeep Holla, Heiko Carstens, Lorenzo Pieralisi,
Greg Kroah-Hartman, linux-api@vger.kernel.org,
linux390@de.ibm.com, linux-arm-kernel@lists.infradead.org,
linux-ia64@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
linux-s390@vger.kernel.org, x86@kernel.org
In-Reply-To: <541CAD16.6030309@codeaurora.org>
Hi Stephen,
On 19/09/14 23:24, Stephen Boyd wrote:
> On 09/03/14 10:00, Sudeep Holla wrote:
>> From: Sudeep Holla <sudeep.holla@arm.com>
>>
>> This patch adds initial support for providing processor cache information
>> to userspace through sysfs interface. This is based on already existing
>> implementations(x86, ia64, s390 and powerpc) and hence the interface is
>> intended to be fully compatible.
>>
>> The main purpose of this generic support is to avoid further code
>> duplication to support new architectures and also to unify all the existing
>> different implementations.
>>
>> This implementation maintains the hierarchy of cache objects which reflects
>> the system's cache topology. Cache devices are instantiated as needed as
>> CPUs come online. The cache information is replicated per-cpu even if they are
>> shared. A per-cpu array of cache information maintained is used mainly for
>> sysfs-related book keeping.
>>
>> It also implements the shared_cpu_map attribute, which is essential for
>> enabling both kernel and user-space to discover the system's overall cache
>> topology.
>>
>> This patch also add the missing ABI documentation for the cacheinfo sysfs
>> interface already, which is well defined and widely used.
>>
>> Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
>> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>> Cc: Stephen Boyd <sboyd@codeaurora.org>
>> Cc: linux-api@vger.kernel.org
>> Cc: linux390@de.ibm.com
>> Cc: linux-arm-kernel@lists.infradead.org
>> Cc: linux-ia64@vger.kernel.org
>> Cc: linuxppc-dev@lists.ozlabs.org
>> Cc: linux-s390@vger.kernel.org
>> Cc: x86@kernel.org
>>
>
> Reviewed-by: Stephen Boyd <sboyd@codeaurora.org>
> Tested-by: Stephen Boyd <sboyd@codeaurora.org>
>
Thanks for all the reviews and testings of the series.
Regards,
Sudeep
^ permalink raw reply
* [PATCH v4 0/5] Add drm driver for Rockchip Socs
From: Mark yao @ 2014-09-22 10:47 UTC (permalink / raw)
To: heiko-4mtYJXux2i+zQB+pC5nmwQ, Boris BREZILLON, David Airlie,
Rob Clark, Rob Herring, Pawel Moll, Mark Rutland, Ian Campbell,
Kumar Gala, Randy Dunlap, Grant Likely, Greg Kroah-Hartman,
John Stultz, Rom Lemarchand
Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-doc-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
dianders-F7+t8E8rja9g9hUCZPvPmw, marcheu-F7+t8E8rja9g9hUCZPvPmw,
dbehr-F7+t8E8rja9g9hUCZPvPmw, olof-nZhT3qVonbNeoWH0uzbU5w,
djkurtz-F7+t8E8rja9g9hUCZPvPmw, xjq-TNX95d0MmH7DzftRWevZcw,
kfx-TNX95d0MmH7DzftRWevZcw, cym-TNX95d0MmH7DzftRWevZcw,
cf-TNX95d0MmH7DzftRWevZcw, zyw-TNX95d0MmH7DzftRWevZcw,
xxm-TNX95d0MmH7DzftRWevZcw, huangtao-TNX95d0MmH7DzftRWevZcw,
kever.yang-TNX95d0MmH7DzftRWevZcw, yxj-TNX95d0MmH7DzftRWevZcw,
wxt-TNX95d0MmH7DzftRWevZcw, xw-TNX95d0MmH7DzftRWevZcw, Mark yao
This a series of patches is a DRM Driver for Rockchip Socs, add support
for vop devices, eDP. Future patches will add additional encoders/connectors,
such as HDMI.
The basic "crtc" for rockchip is a "VOP" - Video Output Processor.
the vop devices found on Rockchip rk3288 Soc, rk3288 soc have two similar
Vop devices. Vop devices support iommu mapping, we use dma-mapping API with
ARM_DMA_USE_IOMMU.
Changes in v2:
- add DRM master device node to list all display nodes that comprise
the graphics subsystem.
- use the component framework to defer main drm driver probe
until all VOP devices have been probed.
- use dma-mapping API with ARM_DMA_USE_IOMMU, create dma mapping by
master device and each vop device can shared the drm dma mapping.
- use drm_crtc_init_with_planes and drm_universal_plane_init.
- remove unnecessary middle layers.
- add cursor set, move funcs to rockchip drm crtc.
- use panel-simple driver for primary display.
- add vop and edp reset.
Changes in v3:
- change "crtc->fb" to "crtc->primary-fb"
Adviced by Daniel Vetter
- init cursor plane with universal api, remove unnecessary cursor set,move
Changes in v4:
Adviced by David Herrmann
- remove drm_platform_*() usage, use register drm device directly.
Adviced by Rob Clark
- remove special mmap ioctl, do userspace mmap with normal mmap() or mmap offset
Tested on rk3288 pinky board, use eDP encoders/connector, boot and display OK
Mark yao (5):
drm/rockchip: Add basic drm driver
dt-bindings: video: Add for rockchip display subsytem
dt-bindings: video: Add documentation for rockchip vop
dt-bindings: video: Add documentation for rockchip edp
drm/rockchip: Add support for Rockchip Soc EDP
.../devicetree/bindings/video/rockchip-drm.txt | 19 +
.../devicetree/bindings/video/rockchip-edp.txt | 50 +
.../devicetree/bindings/video/rockchip-vop.txt | 58 +
drivers/gpu/drm/Kconfig | 2 +
drivers/gpu/drm/Makefile | 1 +
drivers/gpu/drm/rockchip/Kconfig | 28 +
drivers/gpu/drm/rockchip/Makefile | 12 +
drivers/gpu/drm/rockchip/rockchip_drm_drv.c | 524 ++++++++
drivers/gpu/drm/rockchip/rockchip_drm_drv.h | 120 ++
drivers/gpu/drm/rockchip/rockchip_drm_fb.c | 201 +++
drivers/gpu/drm/rockchip/rockchip_drm_fb.h | 28 +
drivers/gpu/drm/rockchip/rockchip_drm_fbdev.c | 231 ++++
drivers/gpu/drm/rockchip/rockchip_drm_fbdev.h | 20 +
drivers/gpu/drm/rockchip/rockchip_drm_gem.c | 404 ++++++
drivers/gpu/drm/rockchip/rockchip_drm_gem.h | 72 +
drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 1372 ++++++++++++++++++++
drivers/gpu/drm/rockchip/rockchip_drm_vop.h | 187 +++
drivers/gpu/drm/rockchip/rockchip_edp_core.c | 853 ++++++++++++
drivers/gpu/drm/rockchip/rockchip_edp_core.h | 309 +++++
drivers/gpu/drm/rockchip/rockchip_edp_reg.c | 1202 +++++++++++++++++
drivers/gpu/drm/rockchip/rockchip_edp_reg.h | 345 +++++
include/uapi/drm/rockchip_drm.h | 75 ++
22 files changed, 6113 insertions(+)
--
1.7.9.5
^ permalink raw reply
* [PATCH v4 1/5] drm/rockchip: Add basic drm driver
From: Mark yao @ 2014-09-22 10:48 UTC (permalink / raw)
To: heiko, Boris BREZILLON, David Airlie, Rob Clark, Rob Herring,
Pawel Moll, Mark Rutland, Ian Campbell, Kumar Gala, Randy Dunlap,
Grant Likely, Greg Kroah-Hartman, John Stultz, Rom Lemarchand
Cc: devicetree, linux-doc, linux-kernel, dri-devel, linux-api,
linux-rockchip, dianders, marcheu, dbehr, olof, djkurtz, xjq, kfx,
cym, cf, zyw, xxm, huangtao, kever.yang, yxj, wxt, xw, Mark yao
In-Reply-To: <1411382820-1615-1-git-send-email-mark.yao@rock-chips.com>
This patch adds the basic structure of a DRM Driver for Rockchip Socs.
Signed-off-by: Mark yao <mark.yao@rock-chips.com>
---
Changes in v2:
- use the component framework to defer main drm driver probe
until all VOP devices have been probed.
- use dma-mapping API with ARM_DMA_USE_IOMMU, create dma mapping by
master device and each vop device can shared the drm dma mapping.
- use drm_crtc_init_with_planes and drm_universal_plane_init.
- remove unnecessary middle layers.
- add cursor set, move funcs to rockchip drm crtc.
- use vop reset at first init
- reference framebuffer when used and unreference when swap out vop
Changes in v3:
- change "crtc->fb" to "crtc->primary-fb"
Adviced by Daniel Vetter
- init cursor plane with universal api, remove unnecessary cursor set,move
Changes in v4:
Adviced by David Herrmann
- remove drm_platform_*() usage, use register drm device directly.
Adviced by Rob Clark
- remove special mmap ioctl, do userspace mmap with normal mmap() or mmap offset
drivers/gpu/drm/Kconfig | 2 +
drivers/gpu/drm/Makefile | 1 +
drivers/gpu/drm/rockchip/Kconfig | 19 +
drivers/gpu/drm/rockchip/Makefile | 10 +
drivers/gpu/drm/rockchip/rockchip_drm_drv.c | 524 ++++++++++
drivers/gpu/drm/rockchip/rockchip_drm_drv.h | 120 +++
drivers/gpu/drm/rockchip/rockchip_drm_fb.c | 201 ++++
drivers/gpu/drm/rockchip/rockchip_drm_fb.h | 28 +
drivers/gpu/drm/rockchip/rockchip_drm_fbdev.c | 231 +++++
drivers/gpu/drm/rockchip/rockchip_drm_fbdev.h | 20 +
drivers/gpu/drm/rockchip/rockchip_drm_gem.c | 404 ++++++++
drivers/gpu/drm/rockchip/rockchip_drm_gem.h | 72 ++
drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 1372 +++++++++++++++++++++++++
drivers/gpu/drm/rockchip/rockchip_drm_vop.h | 187 ++++
include/uapi/drm/rockchip_drm.h | 75 ++
15 files changed, 3266 insertions(+)
create mode 100644 drivers/gpu/drm/rockchip/Kconfig
create mode 100644 drivers/gpu/drm/rockchip/Makefile
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_drv.c
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_drv.h
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_fb.c
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_fb.h
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_fbdev.c
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_fbdev.h
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_gem.c
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_gem.h
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_vop.c
create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_vop.h
create mode 100644 include/uapi/drm/rockchip_drm.h
diff --git a/drivers/gpu/drm/Kconfig b/drivers/gpu/drm/Kconfig
index b066bb3..7c4c3c6 100644
--- a/drivers/gpu/drm/Kconfig
+++ b/drivers/gpu/drm/Kconfig
@@ -171,6 +171,8 @@ config DRM_SAVAGE
source "drivers/gpu/drm/exynos/Kconfig"
+source "drivers/gpu/drm/rockchip/Kconfig"
+
source "drivers/gpu/drm/vmwgfx/Kconfig"
source "drivers/gpu/drm/gma500/Kconfig"
diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
index 4a55d59..d03387a 100644
--- a/drivers/gpu/drm/Makefile
+++ b/drivers/gpu/drm/Makefile
@@ -52,6 +52,7 @@ obj-$(CONFIG_DRM_VMWGFX)+= vmwgfx/
obj-$(CONFIG_DRM_VIA) +=via/
obj-$(CONFIG_DRM_NOUVEAU) +=nouveau/
obj-$(CONFIG_DRM_EXYNOS) +=exynos/
+obj-$(CONFIG_DRM_ROCKCHIP) +=rockchip/
obj-$(CONFIG_DRM_GMA500) += gma500/
obj-$(CONFIG_DRM_UDL) += udl/
obj-$(CONFIG_DRM_AST) += ast/
diff --git a/drivers/gpu/drm/rockchip/Kconfig b/drivers/gpu/drm/rockchip/Kconfig
new file mode 100644
index 0000000..7146c80
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/Kconfig
@@ -0,0 +1,19 @@
+config DRM_ROCKCHIP
+ tristate "DRM Support for Rockchip"
+ depends on DRM && ROCKCHIP_IOMMU
+ select ARM_DMA_USE_IOMMU
+ select IOMMU_API
+ select DRM_KMS_HELPER
+ select DRM_KMS_FB_HELPER
+ select DRM_PANEL
+ select FB_CFB_FILLRECT
+ select FB_CFB_COPYAREA
+ select FB_CFB_IMAGEBLIT
+ select VT_HW_CONSOLE_BINDING if FRAMEBUFFER_CONSOLE
+ select VIDEOMODE_HELPERS
+ help
+ Choose this option if you have a Rockchip soc chipset.
+ This driver provides kernel mode setting and buffer
+ management to userspace. This driver does not provides
+ 2D or 3D acceleration; acceleration is performed by other
+ IP found on the SoC.
diff --git a/drivers/gpu/drm/rockchip/Makefile b/drivers/gpu/drm/rockchip/Makefile
new file mode 100644
index 0000000..6e6d468
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/Makefile
@@ -0,0 +1,10 @@
+#
+# Makefile for the drm device driver. This driver provides support for the
+# Direct Rendering Infrastructure (DRI) in XFree86 4.1.0 and higher.
+
+ccflags-y := -Iinclude/drm -Idrivers/gpu/drm/rockchip
+
+rockchipdrm-y := rockchip_drm_drv.o rockchip_drm_fb.o rockchip_drm_fbdev.o \
+ rockchip_drm_gem.o rockchip_drm_vop.o
+
+obj-$(CONFIG_DRM_ROCKCHIP) += rockchipdrm.o
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_drv.c b/drivers/gpu/drm/rockchip/rockchip_drm_drv.c
new file mode 100644
index 0000000..94926cb
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_drv.c
@@ -0,0 +1,524 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * based on exynos_drm_drv.c
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <asm/dma-iommu.h>
+
+#include <drm/drmP.h>
+#include <drm/drm_crtc_helper.h>
+#include <drm/drm_fb_helper.h>
+#include <linux/dma-mapping.h>
+#include <linux/pm_runtime.h>
+#include <linux/of_graph.h>
+#include <linux/component.h>
+
+#include <drm/rockchip_drm.h>
+
+#include "rockchip_drm_drv.h"
+#include "rockchip_drm_fb.h"
+#include "rockchip_drm_fbdev.h"
+#include "rockchip_drm_gem.h"
+
+#define DRIVER_NAME "rockchip"
+#define DRIVER_DESC "RockChip Soc DRM"
+#define DRIVER_DATE "20140818"
+#define DRIVER_MAJOR 1
+#define DRIVER_MINOR 0
+
+/*
+ * Attach a (component) device to the shared drm dma mapping from master drm
+ * device. This is used by the VOPs to map GEM buffers to a common DMA
+ * mapping.
+ */
+int rockchip_drm_dma_attach_device(struct drm_device *drm_dev,
+ struct device *dev)
+{
+ struct dma_iommu_mapping *mapping = drm_dev->dev->archdata.mapping;
+ int ret;
+
+ ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
+ if (ret)
+ return ret;
+
+ dma_set_max_seg_size(dev, 0xffffffffu);
+
+ return arm_iommu_attach_device(dev, mapping);
+}
+
+void rockchip_drm_dma_detach_device(struct drm_device *drm_dev,
+ struct device *dev)
+{
+ arm_iommu_detach_device(drm_dev->dev);
+}
+
+static int rockchip_drm_load(struct drm_device *drm_dev, unsigned long flags)
+{
+ struct rockchip_drm_private *private;
+ struct dma_iommu_mapping *mapping;
+ struct device *dev = drm_dev->dev;
+ int ret;
+
+ private = devm_kzalloc(drm_dev->dev, sizeof(*private), GFP_KERNEL);
+ if (!private)
+ return -ENOMEM;
+
+ dev_set_drvdata(drm_dev->dev, dev);
+ drm_dev->dev_private = private;
+
+ drm_mode_config_init(drm_dev);
+
+ rockchip_drm_mode_config_init(drm_dev);
+
+ dev->dma_parms = devm_kzalloc(dev, sizeof(*dev->dma_parms),
+ GFP_KERNEL);
+ if (!dev->dma_parms) {
+ ret = -ENOMEM;
+ goto err_config_cleanup;
+ }
+
+ /* TODO(djkurtz): fetch the mapping start/size from somewhere */
+ mapping = arm_iommu_create_mapping(&platform_bus_type, 0x10000000,
+ SZ_1G);
+ if (IS_ERR(mapping)) {
+ ret = PTR_ERR(mapping);
+ goto err_config_cleanup;
+ }
+
+ dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
+ dma_set_max_seg_size(dev, 0xffffffffu);
+
+ ret = arm_iommu_attach_device(dev, mapping);
+ if (ret)
+ goto err_release_mapping;
+
+ /* Try to bind all sub drivers. */
+ ret = component_bind_all(dev, drm_dev);
+ if (ret)
+ goto err_detach_device;
+
+ /* init kms poll for handling hpd */
+ drm_kms_helper_poll_init(drm_dev);
+
+ /*
+ * enable drm irq mode.
+ * - with irq_enabled = true, we can use the vblank feature.
+ */
+ drm_dev->irq_enabled = true;
+
+ /*
+ * with vblank_disable_allowed = true, vblank interrupt will be disabled
+ * by drm timer once a current process gives up ownership of
+ * vblank event.(after drm_vblank_put function is called)
+ */
+ drm_dev->vblank_disable_allowed = true;
+
+ ret = drm_vblank_init(drm_dev, ROCKCHIP_MAX_CRTC);
+ if (ret)
+ goto err_kms_helper_poll_fini;
+
+ rockchip_drm_fbdev_init(drm_dev);
+
+ /* force connectors detection */
+ drm_helper_hpd_irq_event(drm_dev);
+
+ return 0;
+
+err_kms_helper_poll_fini:
+ drm_kms_helper_poll_fini(drm_dev);
+ component_unbind_all(dev, drm_dev);
+err_detach_device:
+ arm_iommu_detach_device(dev);
+err_release_mapping:
+ arm_iommu_release_mapping(dev->archdata.mapping);
+err_config_cleanup:
+ drm_mode_config_cleanup(drm_dev);
+ drm_dev->dev_private = NULL;
+ dev_set_drvdata(dev, NULL);
+ return ret;
+}
+
+static int rockchip_drm_unload(struct drm_device *drm_dev)
+{
+ struct device *dev = drm_dev->dev;
+
+ drm_kms_helper_poll_fini(drm_dev);
+ component_unbind_all(dev, drm_dev);
+ arm_iommu_detach_device(dev);
+ arm_iommu_release_mapping(dev->archdata.mapping);
+ drm_mode_config_cleanup(drm_dev);
+ drm_dev->dev_private = NULL;
+ dev_set_drvdata(dev, NULL);
+
+ return 0;
+}
+
+static int rockchip_drm_suspend(struct drm_device *dev, pm_message_t state)
+{
+ struct drm_connector *connector;
+
+ drm_modeset_lock_all(dev);
+ list_for_each_entry(connector, &dev->mode_config.connector_list, head) {
+ int old_dpms = connector->dpms;
+
+ if (connector->funcs->dpms)
+ connector->funcs->dpms(connector, DRM_MODE_DPMS_OFF);
+
+ /* Set the old mode back to the connector for resume */
+ connector->dpms = old_dpms;
+ }
+ drm_modeset_unlock_all(dev);
+
+ return 0;
+}
+
+static int rockchip_drm_resume(struct drm_device *dev)
+{
+ struct drm_connector *connector;
+
+ drm_modeset_lock_all(dev);
+ list_for_each_entry(connector, &dev->mode_config.connector_list, head) {
+ if (connector->funcs->dpms)
+ connector->funcs->dpms(connector, connector->dpms);
+ }
+ drm_modeset_unlock_all(dev);
+
+ drm_helper_resume_force_mode(dev);
+
+ return 0;
+}
+
+void rockchip_drm_lastclose(struct drm_device *dev)
+{
+ struct rockchip_drm_private *priv = dev->dev_private;
+
+ drm_modeset_lock_all(dev);
+ if (priv->fb_helper)
+ drm_fb_helper_restore_fbdev_mode(priv->fb_helper);
+ drm_modeset_unlock_all(dev);
+}
+
+static const struct drm_ioctl_desc rockchip_ioctls[] = {
+ DRM_IOCTL_DEF_DRV(ROCKCHIP_GEM_CREATE, rockchip_gem_create_ioctl,
+ DRM_UNLOCKED | DRM_AUTH),
+ DRM_IOCTL_DEF_DRV(ROCKCHIP_GEM_GET, rockchip_gem_get_ioctl,
+ DRM_UNLOCKED),
+ DRM_IOCTL_DEF_DRV(ROCKCHIP_GEM_MAP_OFFSET,
+ rockchip_gem_map_offset_ioctl, DRM_UNLOCKED |
+ DRM_AUTH),
+};
+
+static const struct file_operations rockchip_drm_driver_fops = {
+ .owner = THIS_MODULE,
+ .open = drm_open,
+ .mmap = rockchip_drm_gem_mmap,
+ .poll = drm_poll,
+ .read = drm_read,
+ .unlocked_ioctl = drm_ioctl,
+#ifdef CONFIG_COMPAT
+ .compat_ioctl = drm_compat_ioctl,
+#endif
+ .release = drm_release,
+};
+
+const struct vm_operations_struct rockchip_drm_vm_ops = {
+ .open = drm_gem_vm_open,
+ .close = drm_gem_vm_close,
+};
+
+static struct drm_driver rockchip_drm_driver = {
+ .driver_features = DRIVER_MODESET | DRIVER_GEM | DRIVER_PRIME,
+ .load = rockchip_drm_load,
+ .unload = rockchip_drm_unload,
+ .lastclose = rockchip_drm_lastclose,
+ .suspend = rockchip_drm_suspend,
+ .resume = rockchip_drm_resume,
+ .get_vblank_counter = drm_vblank_count,
+ .enable_vblank = rockchip_drm_crtc_enable_vblank,
+ .disable_vblank = rockchip_drm_crtc_disable_vblank,
+ .gem_vm_ops = &rockchip_drm_vm_ops,
+ .gem_free_object = rockchip_gem_free_object,
+ .dumb_create = rockchip_gem_dumb_create,
+ .dumb_map_offset = rockchip_gem_dumb_map_offset,
+ .dumb_destroy = drm_gem_dumb_destroy,
+ .prime_handle_to_fd = drm_gem_prime_handle_to_fd,
+ .prime_fd_to_handle = drm_gem_prime_fd_to_handle,
+ .gem_prime_import = drm_gem_prime_import,
+ .gem_prime_export = drm_gem_prime_export,
+ .gem_prime_get_sg_table = rockchip_gem_prime_get_sg_table,
+ .gem_prime_import_sg_table = rockchip_gem_prime_import_sg_table,
+ .gem_prime_vmap = rockchip_gem_prime_vmap,
+ .gem_prime_vunmap = rockchip_gem_prime_vunmap,
+ .gem_prime_mmap = rockchip_gem_prime_mmap,
+ .ioctls = rockchip_ioctls,
+ .num_ioctls = ARRAY_SIZE(rockchip_ioctls),
+ .fops = &rockchip_drm_driver_fops,
+ .name = DRIVER_NAME,
+ .desc = DRIVER_DESC,
+ .date = DRIVER_DATE,
+ .major = DRIVER_MAJOR,
+ .minor = DRIVER_MINOR,
+};
+
+#ifdef CONFIG_PM_SLEEP
+static int rockchip_drm_sys_suspend(struct device *dev)
+{
+ struct drm_device *drm_dev = dev_get_drvdata(dev);
+ pm_message_t message;
+
+ if (pm_runtime_suspended(dev))
+ return 0;
+
+ message.event = PM_EVENT_SUSPEND;
+
+ return rockchip_drm_suspend(drm_dev, message);
+}
+
+static int rockchip_drm_sys_resume(struct device *dev)
+{
+ struct drm_device *drm_dev = dev_get_drvdata(dev);
+
+ if (pm_runtime_suspended(dev))
+ return 0;
+
+ return rockchip_drm_resume(drm_dev);
+}
+#endif
+
+static const struct dev_pm_ops rockchip_drm_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(rockchip_drm_sys_suspend,
+ rockchip_drm_sys_resume)
+};
+
+int rockchip_drm_add_crtc(struct drm_device *drm, struct drm_crtc *crtc,
+ struct device_node *np)
+{
+ struct rockchip_drm_private *priv = drm->dev_private;
+ struct device_node *port;
+ int pipe;
+
+ if (priv->num_pipe >= ROCKCHIP_MAX_CRTC)
+ return -EINVAL;
+
+ port = of_get_child_by_name(np, "port");
+ of_node_put(np);
+ if (!port) {
+ dev_err(drm->dev, "no port node found in %s\n",
+ np->full_name);
+ return -ENXIO;
+ }
+ pipe = priv->num_pipe++;
+ crtc->port = port;
+
+ priv->crtc[pipe] = crtc;
+
+ return pipe;
+}
+
+void rockchip_drm_remove_crtc(struct drm_device *drm, int pipe)
+{
+ struct rockchip_drm_private *priv = drm->dev_private;
+
+ priv->num_pipe--;
+ of_node_put(priv->crtc[pipe]->port);
+ priv->crtc[pipe] = NULL;
+}
+
+struct drm_crtc *rockchip_find_crtc(struct drm_device *drm, int pipe)
+{
+ struct rockchip_drm_private *priv = drm->dev_private;
+
+ if (pipe < ROCKCHIP_MAX_CRTC && priv->crtc[pipe])
+ return priv->crtc[pipe];
+
+ return NULL;
+}
+
+/*
+ * @node: device tree node containing encoder input ports
+ * @encoder: drm_encoder
+ */
+int rockchip_drm_encoder_get_mux_id(struct device_node *node,
+ struct drm_encoder *encoder)
+{
+ struct device_node *ep = NULL;
+ struct drm_crtc *crtc = encoder->crtc;
+ struct of_endpoint endpoint;
+ struct device_node *port;
+ int ret;
+
+ if (!node || !crtc)
+ return -EINVAL;
+
+ do {
+ ep = of_graph_get_next_endpoint(node, ep);
+ if (!ep)
+ break;
+
+ port = of_graph_get_remote_port(ep);
+ of_node_put(port);
+ if (port == crtc->port) {
+ ret = of_graph_parse_endpoint(ep, &endpoint);
+ return ret ? ret : endpoint.id;
+ }
+ } while (ep);
+
+ return -EINVAL;
+}
+
+static int compare_of(struct device *dev, void *data)
+{
+ struct device_node *np = data;
+
+ return dev->of_node == np;
+}
+
+static void rockchip_add_endpoints(struct device *dev,
+ struct component_match **match,
+ struct device_node *port)
+{
+ struct device_node *ep, *remote;
+
+ for_each_child_of_node(port, ep) {
+ remote = of_graph_get_remote_port_parent(ep);
+ if (!remote || !of_device_is_available(remote)) {
+ of_node_put(remote);
+ continue;
+ } else if (!of_device_is_available(remote->parent)) {
+ dev_warn(dev, "parent device of %s is not available\n",
+ remote->full_name);
+ of_node_put(remote);
+ continue;
+ }
+
+ component_match_add(dev, match, compare_of, remote);
+ of_node_put(remote);
+ }
+}
+
+static int rockchip_drm_bind(struct device *dev)
+{
+ struct drm_device *drm;
+ int ret;
+
+ drm = drm_dev_alloc(&rockchip_drm_driver, dev);
+ if (!drm)
+ return -ENOMEM;
+
+ ret = drm_dev_set_unique(drm, "%s", dev_name(dev));
+ if (ret)
+ goto err_free;
+
+ ret = drm_dev_register(drm, 0);
+ if (ret)
+ goto err_free;
+
+ dev_set_drvdata(dev, drm);
+
+ return 0;
+
+err_free:
+ drm_dev_unref(drm);
+ return ret;
+}
+
+static void rockchip_drm_unbind(struct device *dev)
+{
+ struct drm_device *drm = dev_get_drvdata(dev);
+
+ drm_dev_unregister(drm);
+ drm_dev_unref(drm);
+}
+
+static const struct component_master_ops rockchip_drm_ops = {
+ .bind = rockchip_drm_bind,
+ .unbind = rockchip_drm_unbind,
+};
+
+static int rockchip_drm_platform_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct component_match *match = NULL;
+ struct device_node *np = dev->of_node;
+ struct device_node *port;
+ int i;
+ int ret;
+
+ if (!np)
+ return -ENODEV;
+ /*
+ * Bind the crtc ports first, so that
+ * drm_of_find_possible_crtcs called from encoder .bind callbacks
+ * works as expected.
+ */
+ for (i = 0;; i++) {
+ port = of_parse_phandle(np, "ports", i);
+ if (!port)
+ break;
+
+ component_match_add(dev, &match, compare_of, port->parent);
+ of_node_put(port);
+ }
+
+ if (i == 0) {
+ dev_err(dev, "missing 'ports' property\n");
+ return -ENODEV;
+ }
+ /*
+ * For each bound crtc, bind the encoders attached to its
+ * remote endpoint.
+ */
+ for (i = 0;; i++) {
+ port = of_parse_phandle(np, "ports", i);
+ if (!port)
+ break;
+
+ rockchip_add_endpoints(dev, &match, port);
+ of_node_put(port);
+ }
+
+ ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
+ if (ret)
+ return ret;
+
+ return component_master_add_with_match(dev, &rockchip_drm_ops, match);
+}
+
+static int rockchip_drm_platform_remove(struct platform_device *pdev)
+{
+ component_master_del(&pdev->dev, &rockchip_drm_ops);
+ return 0;
+}
+
+static const struct of_device_id rockchip_drm_dt_ids[] = {
+ { .compatible = "rockchip,display-subsystem", },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, rockchip_drm_dt_ids);
+
+static struct platform_driver rockchip_drm_platform_driver = {
+ .probe = rockchip_drm_platform_probe,
+ .remove = rockchip_drm_platform_remove,
+ .driver = {
+ .owner = THIS_MODULE,
+ .name = "rockchip-drm",
+ .of_match_table = rockchip_drm_dt_ids,
+ },
+};
+
+module_platform_driver(rockchip_drm_platform_driver);
+
+MODULE_AUTHOR("Mark Yao <mark.yao@rock-chips.com>");
+MODULE_DESCRIPTION("ROCKCHIP DRM Driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_drv.h b/drivers/gpu/drm/rockchip/rockchip_drm_drv.h
new file mode 100644
index 0000000..154b3ec
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_drv.h
@@ -0,0 +1,120 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * based on exynos_drm_drv.h
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _ROCKCHIP_DRM_DRV_H
+#define _ROCKCHIP_DRM_DRV_H
+
+#include <linux/module.h>
+#include <linux/component.h>
+
+#define ROCKCHIP_MAX_FB_BUFFER 4
+#define ROCKCHIP_MAX_CONNECTOR 2
+
+struct drm_device;
+struct drm_connector;
+
+/*
+ * display output interface supported by rockchip lcdc
+ */
+#define ROCKCHIP_OUTFACE_P888 0
+#define ROCKCHIP_OUTFACE_P666 1
+#define ROCKCHIP_OUTFACE_P565 2
+/* for use special outface */
+#define ROCKCHIP_OUTFACE_AAAA 15
+
+#define ROCKCHIP_COLOR_SWAP_RG 0x1
+#define ROCKCHIP_COLOR_SWAP_RB 0x2
+#define ROCKCHIP_COLOR_SWAP_GB 0x4
+
+/*
+ * Special mode info for rockchip
+ *
+ * @out_type: lcd controller need to know the sceen type.
+ */
+struct rockchip_display_mode {
+ int out_type;
+};
+
+#define ROCKCHIP_EVENT_HOTPLUG 1
+
+enum rockchip_plane_type {
+ ROCKCHIP_WIN0,
+ ROCKCHIP_WIN1,
+ ROCKCHIP_WIN2,
+ ROCKCHIP_WIN3,
+ ROCKCHIP_CURSOR,
+ ROCKCHIP_MAX_PLANE,
+};
+
+/* This enumerates device type. */
+enum rockchip_drm_device_type {
+ ROCKCHIP_DEVICE_TYPE_NONE,
+ ROCKCHIP_DEVICE_TYPE_CRTC,
+ ROCKCHIP_DEVICE_TYPE_CONNECTOR,
+};
+
+/* this enumerates display type. */
+enum rockchip_drm_output_type {
+ ROCKCHIP_DISPLAY_TYPE_NONE = 0,
+ /* RGB Interface. */
+ ROCKCHIP_DISPLAY_TYPE_RGB,
+ /* LVDS Interface. */
+ ROCKCHIP_DISPLAY_TYPE_LVDS,
+ /* EDP Interface. */
+ ROCKCHIP_DISPLAY_TYPE_EDP,
+ /* MIPI Interface. */
+ ROCKCHIP_DISPLAY_TYPE_MIPI,
+ /* HDMI Interface. */
+ ROCKCHIP_DISPLAY_TYPE_HDMI,
+};
+
+enum rockchip_crtc_type {
+ ROCKCHIP_CRTC_VOPB,
+ ROCKCHIP_CRTC_VOPL,
+ ROCKCHIP_MAX_CRTC,
+};
+
+/*
+ * Rockchip drm private structure.
+ *
+ * @num_pipe: number of pipes for this device.
+ */
+struct rockchip_drm_private {
+ struct drm_fb_helper *fb_helper;
+ /*
+ * created crtc object would be contained at this array and
+ * this array is used to be aware of which crtc did it request vblank.
+ */
+ struct drm_crtc *crtc[ROCKCHIP_MAX_CRTC];
+
+ unsigned int num_pipe;
+};
+
+int rockchip_drm_add_crtc(struct drm_device *drm, struct drm_crtc *crtc,
+ struct device_node *port);
+void rockchip_drm_remove_crtc(struct drm_device *drm, int pipe);
+struct drm_crtc *rockchip_find_crtc(struct drm_device *drm, int pipe);
+int rockchip_drm_encoder_get_mux_id(struct device_node *node,
+ struct drm_encoder *encoder);
+void rockchip_drm_crtc_finish_pageflip(struct drm_device *dev, int pipe);
+void rockchip_drm_crtc_cancel_pending_flip(struct drm_device *dev);
+int rockchip_drm_crtc_enable_vblank(struct drm_device *dev, int pipe);
+void rockchip_drm_crtc_disable_vblank(struct drm_device *dev, int pipe);
+int rockchip_drm_dma_attach_device(struct drm_device *drm_dev,
+ struct device *dev);
+void rockchip_drm_dma_detach_device(struct drm_device *drm_dev,
+ struct device *dev);
+#endif /* _ROCKCHIP_DRM_DRV_H_ */
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_fb.c b/drivers/gpu/drm/rockchip/rockchip_drm_fb.c
new file mode 100644
index 0000000..b319505
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_fb.c
@@ -0,0 +1,201 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <drm/drmP.h>
+#include <drm/drm_fb_helper.h>
+#include <drm/drm_crtc_helper.h>
+#include <uapi/drm/rockchip_drm.h>
+
+#include "rockchip_drm_drv.h"
+#include "rockchip_drm_gem.h"
+
+#define to_rockchip_fb(x) container_of(x, struct rockchip_drm_fb, fb)
+
+struct rockchip_drm_fb {
+ struct drm_framebuffer fb;
+ struct drm_gem_object *obj[ROCKCHIP_MAX_FB_BUFFER];
+};
+
+struct drm_gem_object *rockchip_fb_get_gem_obj(struct drm_framebuffer *fb,
+ unsigned int plane)
+{
+ struct rockchip_drm_fb *rk_fb = to_rockchip_fb(fb);
+
+ if (plane >= ROCKCHIP_MAX_FB_BUFFER)
+ return NULL;
+
+ return rk_fb->obj[plane];
+}
+
+static void rockchip_drm_fb_destroy(struct drm_framebuffer *fb)
+{
+ struct rockchip_drm_fb *rockchip_fb = to_rockchip_fb(fb);
+ struct drm_gem_object *obj;
+ int i;
+
+ for (i = 0; i < ROCKCHIP_MAX_FB_BUFFER; i++) {
+ obj = rockchip_fb->obj[i];
+ if (obj)
+ drm_gem_object_unreference_unlocked(obj);
+ }
+
+ drm_framebuffer_cleanup(fb);
+ kfree(rockchip_fb);
+}
+
+static int rockchip_drm_fb_create_handle(struct drm_framebuffer *fb,
+ struct drm_file *file_priv,
+ unsigned int *handle)
+{
+ struct rockchip_drm_fb *rockchip_fb = to_rockchip_fb(fb);
+
+ return drm_gem_handle_create(file_priv,
+ rockchip_fb->obj[0], handle);
+}
+
+static struct drm_framebuffer_funcs rockchip_drm_fb_funcs = {
+ .destroy = rockchip_drm_fb_destroy,
+ .create_handle = rockchip_drm_fb_create_handle,
+};
+
+static struct rockchip_drm_fb *
+rockchip_fb_alloc(struct drm_device *dev, struct drm_mode_fb_cmd2 *mode_cmd,
+ struct drm_gem_object **obj, unsigned int num_planes)
+{
+ struct rockchip_drm_fb *rockchip_fb;
+ int ret;
+ int i;
+
+ rockchip_fb = kzalloc(sizeof(*rockchip_fb), GFP_KERNEL);
+ if (!rockchip_fb)
+ return ERR_PTR(-ENOMEM);
+
+ drm_helper_mode_fill_fb_struct(&rockchip_fb->fb, mode_cmd);
+
+ for (i = 0; i < num_planes; i++)
+ rockchip_fb->obj[i] = obj[i];
+
+ ret = drm_framebuffer_init(dev, &rockchip_fb->fb,
+ &rockchip_drm_fb_funcs);
+ if (ret) {
+ dev_err(dev->dev, "Failed to initialize framebuffer: %d\n",
+ ret);
+ kfree(rockchip_fb);
+ return ERR_PTR(ret);
+ }
+
+ return rockchip_fb;
+}
+
+static struct drm_framebuffer *
+rockchip_user_fb_create(struct drm_device *dev, struct drm_file *file_priv,
+ struct drm_mode_fb_cmd2 *mode_cmd)
+{
+ struct rockchip_drm_fb *rockchip_fb;
+ struct drm_gem_object *objs[ROCKCHIP_MAX_FB_BUFFER];
+ struct drm_gem_object *obj;
+ unsigned int hsub;
+ unsigned int vsub;
+ int num_planes;
+ int ret;
+ int i;
+
+ hsub = drm_format_horz_chroma_subsampling(mode_cmd->pixel_format);
+ vsub = drm_format_vert_chroma_subsampling(mode_cmd->pixel_format);
+ num_planes = min(drm_format_num_planes(mode_cmd->pixel_format),
+ ROCKCHIP_MAX_FB_BUFFER);
+
+ for (i = 0; i < num_planes; i++) {
+ unsigned int width = mode_cmd->width / (i ? hsub : 1);
+ unsigned int height = mode_cmd->height / (i ? vsub : 1);
+ unsigned int min_size;
+
+ obj = drm_gem_object_lookup(dev, file_priv,
+ mode_cmd->handles[i]);
+ if (!obj) {
+ dev_err(dev->dev, "Failed to lookup GEM object\n");
+ ret = -ENXIO;
+ goto err_gem_object_unreference;
+ }
+
+ min_size = (height - 1) * mode_cmd->pitches[i] +
+ mode_cmd->offsets[i] +
+ width * drm_format_plane_cpp(mode_cmd->pixel_format, i);
+
+ if (obj->size < min_size) {
+ drm_gem_object_unreference_unlocked(obj);
+ ret = -EINVAL;
+ goto err_gem_object_unreference;
+ }
+ objs[i] = obj;
+ }
+
+ rockchip_fb = rockchip_fb_alloc(dev, mode_cmd, objs, i);
+ if (IS_ERR(rockchip_fb)) {
+ ret = PTR_ERR(rockchip_fb);
+ goto err_gem_object_unreference;
+ }
+
+ return &rockchip_fb->fb;
+
+err_gem_object_unreference:
+ for (i--; i >= 0; i--)
+ drm_gem_object_unreference_unlocked(objs[i]);
+ return ERR_PTR(ret);
+}
+
+static void rockchip_drm_output_poll_changed(struct drm_device *dev)
+{
+ struct rockchip_drm_private *private = dev->dev_private;
+ struct drm_fb_helper *fb_helper = private->fb_helper;
+
+ if (fb_helper)
+ drm_fb_helper_hotplug_event(fb_helper);
+}
+
+static const struct drm_mode_config_funcs rockchip_drm_mode_config_funcs = {
+ .fb_create = rockchip_user_fb_create,
+ .output_poll_changed = rockchip_drm_output_poll_changed,
+};
+
+struct drm_framebuffer *
+rockchip_drm_framebuffer_init(struct drm_device *dev,
+ struct drm_mode_fb_cmd2 *mode_cmd,
+ struct drm_gem_object *obj)
+{
+ struct rockchip_drm_fb *rockchip_fb;
+
+ rockchip_fb = rockchip_fb_alloc(dev, mode_cmd, &obj, 1);
+ if (IS_ERR(rockchip_fb))
+ return NULL;
+
+ return &rockchip_fb->fb;
+}
+
+void rockchip_drm_mode_config_init(struct drm_device *dev)
+{
+ dev->mode_config.min_width = 0;
+ dev->mode_config.min_height = 0;
+
+ /*
+ * set max width and height as default value(4096x4096).
+ * this value would be used to check framebuffer size limitation
+ * at drm_mode_addfb().
+ */
+ dev->mode_config.max_width = 4096;
+ dev->mode_config.max_height = 4096;
+
+ dev->mode_config.funcs = &rockchip_drm_mode_config_funcs;
+}
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_fb.h b/drivers/gpu/drm/rockchip/rockchip_drm_fb.h
new file mode 100644
index 0000000..09574d4
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_fb.h
@@ -0,0 +1,28 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _ROCKCHIP_DRM_FB_H
+#define _ROCKCHIP_DRM_FB_H
+
+struct drm_framebuffer *
+rockchip_drm_framebuffer_init(struct drm_device *dev,
+ struct drm_mode_fb_cmd2 *mode_cmd,
+ struct drm_gem_object *obj);
+void rockchip_drm_framebuffer_fini(struct drm_framebuffer *fb);
+
+void rockchip_drm_mode_config_init(struct drm_device *dev);
+
+struct drm_gem_object *rockchip_fb_get_gem_obj(struct drm_framebuffer *fb,
+ unsigned int plane);
+#endif /* _ROCKCHIP_DRM_FB_H */
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_fbdev.c b/drivers/gpu/drm/rockchip/rockchip_drm_fbdev.c
new file mode 100644
index 0000000..fe1bb22
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_fbdev.c
@@ -0,0 +1,231 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <drm/drmP.h>
+#include <drm/drm_fb_helper.h>
+#include <drm/drm_crtc_helper.h>
+
+#include <drm/rockchip_drm.h>
+
+#include "rockchip_drm_drv.h"
+#include "rockchip_drm_gem.h"
+#include "rockchip_drm_fb.h"
+
+#define PREFERRED_BPP 32
+#define to_rockchip_fbdev(x) container_of(x, struct rockchip_fbdev, helper)
+
+struct rockchip_fbdev {
+ struct drm_fb_helper helper;
+ struct drm_gem_object *bo;
+};
+
+static int rockchip_fbdev_mmap(struct fb_info *info,
+ struct vm_area_struct *vma)
+{
+ struct drm_fb_helper *helper = info->par;
+ struct rockchip_fbdev *fbdev = to_rockchip_fbdev(helper);
+
+ return rockchip_gem_mmap(fbdev->bo, vma);
+}
+
+static struct fb_ops rockchip_drm_fbdev_ops = {
+ .owner = THIS_MODULE,
+ .fb_mmap = rockchip_fbdev_mmap,
+ .fb_fillrect = cfb_fillrect,
+ .fb_copyarea = cfb_copyarea,
+ .fb_imageblit = cfb_imageblit,
+ .fb_check_var = drm_fb_helper_check_var,
+ .fb_set_par = drm_fb_helper_set_par,
+ .fb_blank = drm_fb_helper_blank,
+ .fb_pan_display = drm_fb_helper_pan_display,
+ .fb_setcmap = drm_fb_helper_setcmap,
+};
+
+static int rockchip_drm_fbdev_create(struct drm_fb_helper *helper,
+ struct drm_fb_helper_surface_size *sizes)
+{
+ struct rockchip_fbdev *fbdev = to_rockchip_fbdev(helper);
+ struct drm_mode_fb_cmd2 mode_cmd = { 0 };
+ struct drm_device *dev = helper->dev;
+ struct rockchip_gem_object *rk_obj;
+ struct drm_framebuffer *fb;
+ unsigned int bytes_per_pixel;
+ unsigned long offset;
+ struct fb_info *fbi;
+ size_t size;
+ int ret;
+
+ bytes_per_pixel = DIV_ROUND_UP(sizes->surface_bpp, 8);
+
+ mode_cmd.width = sizes->surface_width;
+ mode_cmd.height = sizes->surface_height;
+ mode_cmd.pitches[0] = sizes->surface_width * bytes_per_pixel;
+ mode_cmd.pixel_format = drm_mode_legacy_fb_format(sizes->surface_bpp,
+ sizes->surface_depth);
+
+ size = mode_cmd.pitches[0] * mode_cmd.height;
+
+ rk_obj = rockchip_gem_create_object(dev, size);
+ if (IS_ERR(rk_obj))
+ return -ENOMEM;
+
+ fbdev->bo = &rk_obj->base;
+
+ fbi = framebuffer_alloc(0, dev->dev);
+ if (!fbi) {
+ dev_err(dev->dev, "Failed to allocate framebuffer info.\n");
+ ret = -ENOMEM;
+ goto err_rockchip_gem_free_object;
+ }
+
+ helper->fb = rockchip_drm_framebuffer_init(dev, &mode_cmd, fbdev->bo);
+ if (IS_ERR(helper->fb)) {
+ dev_err(dev->dev, "Failed to allocate DRM framebuffer.\n");
+ ret = PTR_ERR(helper->fb);
+ goto err_framebuffer_release;
+ }
+
+ helper->fbdev = fbi;
+
+ fbi->par = helper;
+ fbi->flags = FBINFO_FLAG_DEFAULT;
+ fbi->fbops = &rockchip_drm_fbdev_ops;
+
+ ret = fb_alloc_cmap(&fbi->cmap, 256, 0);
+ if (ret) {
+ dev_err(dev->dev, "Failed to allocate color map.\n");
+ goto err_drm_framebuffer_unref;
+ }
+
+ fb = helper->fb;
+ drm_fb_helper_fill_fix(fbi, fb->pitches[0], fb->depth);
+ drm_fb_helper_fill_var(fbi, helper, fb->width, fb->height);
+
+ offset = fbi->var.xoffset * bytes_per_pixel;
+ offset += fbi->var.yoffset * fb->pitches[0];
+
+ dev->mode_config.fb_base = 0;
+ fbi->screen_base = rk_obj->kvaddr + offset;
+ fbi->screen_size = rk_obj->base.size;
+ fbi->fix.smem_len = rk_obj->base.size;
+
+ DRM_DEBUG_KMS("FB [%dx%d]-%d kvaddr=%p offset=%ld size=%d\n",
+ fb->width, fb->height, fb->depth, rk_obj->kvaddr,
+ offset, size);
+ return 0;
+
+err_drm_framebuffer_unref:
+ drm_framebuffer_unreference(helper->fb);
+err_framebuffer_release:
+ framebuffer_release(fbi);
+err_rockchip_gem_free_object:
+ rockchip_gem_free_object(&rk_obj->base);
+ return ret;
+}
+
+static struct drm_fb_helper_funcs rockchip_drm_fb_helper_funcs = {
+ .fb_probe = rockchip_drm_fbdev_create,
+};
+
+int rockchip_drm_fbdev_init(struct drm_device *dev)
+{
+ struct rockchip_drm_private *private = dev->dev_private;
+ struct rockchip_fbdev *fbdev;
+ struct drm_fb_helper *helper;
+ unsigned int num_crtc;
+ int ret;
+
+ if (!dev->mode_config.num_crtc || !dev->mode_config.num_connector)
+ return -EINVAL;
+
+ if (private->fb_helper) {
+ DRM_ERROR("no allow to reinit fbdev\n");
+ return -EINVAL;
+ }
+
+ num_crtc = dev->mode_config.num_crtc;
+
+ fbdev = kzalloc(sizeof(*fbdev), GFP_KERNEL);
+ if (!fbdev)
+ return -ENOMEM;
+
+ fbdev->helper.funcs = &rockchip_drm_fb_helper_funcs;
+ helper = &fbdev->helper;
+
+ ret = drm_fb_helper_init(dev, helper, num_crtc, ROCKCHIP_MAX_CONNECTOR);
+ if (ret < 0) {
+ dev_err(dev->dev, "Failed to initialize drm fb helper.\n");
+ goto err_free;
+ }
+
+ ret = drm_fb_helper_single_add_all_connectors(helper);
+ if (ret < 0) {
+ dev_err(dev->dev, "Failed to add connectors.\n");
+ goto err_drm_fb_helper_fini;
+ }
+
+ /* disable all the possible outputs/crtcs before entering KMS mode */
+ drm_helper_disable_unused_functions(dev);
+
+ ret = drm_fb_helper_initial_config(helper, PREFERRED_BPP);
+ if (ret < 0) {
+ dev_err(dev->dev, "Failed to set initial hw configuration.\n");
+ goto err_drm_fb_helper_fini;
+ }
+
+ private->fb_helper = helper;
+
+ return 0;
+
+err_drm_fb_helper_fini:
+ drm_fb_helper_fini(helper);
+err_free:
+ kfree(fbdev);
+ return ret;
+}
+
+void rockchip_drm_fbdev_fini(struct drm_device *dev)
+{
+ struct rockchip_drm_private *private = dev->dev_private;
+ struct drm_fb_helper *helper;
+ struct rockchip_fbdev *fbdev;
+
+ if (!private || !private->fb_helper)
+ return;
+
+ helper = private->fb_helper;
+ fbdev = to_rockchip_fbdev(helper);
+
+ if (helper->fbdev) {
+ struct fb_info *info;
+ int ret;
+
+ info = helper->fbdev;
+ ret = unregister_framebuffer(info);
+ if (ret < 0)
+ DRM_DEBUG_KMS("failed unregister_framebuffer()\n");
+
+ if (info->cmap.len)
+ fb_dealloc_cmap(&info->cmap);
+
+ framebuffer_release(info);
+ }
+
+ if (helper->fb)
+ drm_framebuffer_unreference(helper->fb);
+
+ drm_fb_helper_fini(helper);
+ kfree(fbdev);
+ private->fb_helper = NULL;
+}
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_fbdev.h b/drivers/gpu/drm/rockchip/rockchip_drm_fbdev.h
new file mode 100644
index 0000000..5edcf6a
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_fbdev.h
@@ -0,0 +1,20 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _ROCKCHIP_DRM_FBDEV_H
+#define _ROCKCHIP_DRM_FBDEV_H
+
+int rockchip_drm_fbdev_init(struct drm_device *dev);
+
+#endif /* _ROCKCHIP_DRM_FBDEV_H */
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_gem.c b/drivers/gpu/drm/rockchip/rockchip_drm_gem.c
new file mode 100644
index 0000000..2f34e92
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_gem.c
@@ -0,0 +1,404 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <drm/drmP.h>
+#include <drm/drm_vma_manager.h>
+#include <drm/rockchip_drm.h>
+
+#include <linux/anon_inodes.h>
+#include <linux/dma-attrs.h>
+
+#include "rockchip_drm_drv.h"
+#include "rockchip_drm_gem.h"
+
+static int rockchip_gem_alloc_buf(struct rockchip_gem_object *rk_obj)
+{
+ struct drm_gem_object *obj = &rk_obj->base;
+ struct drm_device *drm = obj->dev;
+
+ init_dma_attrs(&rk_obj->dma_attrs);
+ dma_set_attr(DMA_ATTR_WRITE_COMBINE, &rk_obj->dma_attrs);
+
+ /* TODO(djkurtz): Use DMA_ATTR_NO_KERNEL_MAPPING except for fbdev */
+ rk_obj->kvaddr = dma_alloc_attrs(drm->dev, obj->size,
+ &rk_obj->dma_addr, GFP_KERNEL,
+ &rk_obj->dma_attrs);
+ if (IS_ERR(rk_obj->kvaddr)) {
+ int ret = PTR_ERR(rk_obj->kvaddr);
+
+ DRM_ERROR("failed to allocate %#x byte dma buffer, %d",
+ obj->size, ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void rockchip_gem_free_buf(struct rockchip_gem_object *rk_obj)
+{
+ struct drm_gem_object *obj = &rk_obj->base;
+ struct drm_device *drm = obj->dev;
+
+ dma_free_attrs(drm->dev, obj->size, rk_obj->kvaddr, rk_obj->dma_addr,
+ &rk_obj->dma_attrs);
+}
+
+/* drm driver mmap file operations */
+int rockchip_drm_gem_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ struct drm_file *priv = filp->private_data;
+ struct drm_device *dev = priv->minor->dev;
+ struct drm_gem_object *obj;
+ struct drm_vma_offset_node *node;
+ int ret;
+
+ if (drm_device_is_unplugged(dev))
+ return -ENODEV;
+
+ mutex_lock(&dev->struct_mutex);
+
+ node = drm_vma_offset_exact_lookup(dev->vma_offset_manager,
+ vma->vm_pgoff,
+ vma_pages(vma));
+ if (!node) {
+ mutex_unlock(&dev->struct_mutex);
+ DRM_ERROR("failed to find vma node.\n");
+ return -EINVAL;
+ } else if (!drm_vma_node_is_allowed(node, filp)) {
+ mutex_unlock(&dev->struct_mutex);
+ return -EACCES;
+ }
+
+ obj = container_of(node, struct drm_gem_object, vma_node);
+ ret = rockchip_gem_mmap(obj, vma);
+
+ mutex_unlock(&dev->struct_mutex);
+
+ return ret;
+}
+
+int rockchip_drm_gem_mmap_buffer(struct file *filp,
+ struct vm_area_struct *vma)
+{
+ struct drm_gem_object *obj = filp->private_data;
+
+ return rockchip_gem_mmap(obj, vma);
+}
+
+static const struct file_operations rockchip_drm_gem_fops = {
+ .mmap = rockchip_drm_gem_mmap_buffer,
+};
+
+struct rockchip_gem_object *
+ rockchip_gem_create_object(struct drm_device *drm, unsigned int size)
+{
+ struct rockchip_gem_object *rk_obj;
+ struct drm_gem_object *obj;
+ struct file *filp;
+ int ret;
+
+ size = round_up(size, PAGE_SIZE);
+
+ rk_obj = kzalloc(sizeof(*rk_obj), GFP_KERNEL);
+ if (!rk_obj)
+ return ERR_PTR(-ENOMEM);
+
+ obj = &rk_obj->base;
+
+ drm_gem_private_object_init(drm, obj, size);
+
+ filp = anon_inode_getfile("rockchip_gem", &rockchip_drm_gem_fops,
+ obj, 0);
+ if (IS_ERR(filp)) {
+ DRM_ERROR("failed to create anon file object.\n");
+ ret = PTR_ERR(filp);
+ goto err_free_rk_obj;
+ }
+ filp->f_mode = FMODE_READ | FMODE_WRITE;
+ obj->filp = filp;
+
+ ret = drm_gem_create_mmap_offset(obj);
+ if (ret)
+ goto err_free_obj;
+
+ ret = rockchip_gem_alloc_buf(rk_obj);
+ if (ret)
+ goto err_free_mmap_offset;
+
+ return rk_obj;
+
+err_free_mmap_offset:
+ drm_gem_free_mmap_offset(obj);
+err_free_obj:
+ drm_gem_object_release(obj);
+err_free_rk_obj:
+ kfree(rk_obj);
+ return ERR_PTR(ret);
+}
+
+/*
+ * rockchip_gem_free_object - (struct drm_driver)->gem_free_object callback
+ * function
+ */
+void rockchip_gem_free_object(struct drm_gem_object *obj)
+{
+ struct rockchip_gem_object *rk_obj;
+
+ drm_gem_free_mmap_offset(obj);
+
+ rk_obj = to_rockchip_obj(obj);
+
+ rockchip_gem_free_buf(rk_obj);
+ drm_gem_free_mmap_offset(obj);
+
+ drm_gem_object_release(obj);
+
+ kfree(rk_obj);
+}
+
+int rockchip_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma)
+{
+ struct rockchip_gem_object *rk_obj = to_rockchip_obj(obj);
+ struct drm_device *drm = obj->dev;
+ unsigned long vm_size;
+
+ vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP;
+ vm_size = vma->vm_end - vma->vm_start;
+
+ if (vm_size > obj->size)
+ return -EINVAL;
+
+ return dma_mmap_attrs(drm->dev, vma, rk_obj->kvaddr, rk_obj->dma_addr,
+ obj->size, &rk_obj->dma_attrs);
+}
+
+/*
+ * rockchip_gem_create_with_handle - allocate an object with the given
+ * size and create a gem handle on it
+ *
+ * returns a struct rockchip_gem_object* on success or ERR_PTR values
+ * on failure.
+ */
+static struct rockchip_gem_object *
+rockchip_gem_create_with_handle(struct drm_file *file_priv,
+ struct drm_device *drm, unsigned int size,
+ unsigned int *handle)
+{
+ struct rockchip_gem_object *rk_obj;
+ struct drm_gem_object *obj;
+ int ret;
+
+ rk_obj = rockchip_gem_create_object(drm, size);
+ if (IS_ERR(rk_obj))
+ return NULL;
+
+ obj = &rk_obj->base;
+
+ /*
+ * allocate a id of idr table where the obj is registered
+ * and handle has the id what user can see.
+ */
+ ret = drm_gem_handle_create(file_priv, obj, handle);
+ if (ret)
+ goto err_handle_create;
+
+ /* drop reference from allocate - handle holds it now. */
+ drm_gem_object_unreference_unlocked(obj);
+
+ return rk_obj;
+
+err_handle_create:
+ rockchip_gem_free_object(obj);
+
+ return ERR_PTR(ret);
+}
+
+int rockchip_gem_dumb_map_offset(struct drm_file *file_priv,
+ struct drm_device *dev, uint32_t handle,
+ uint64_t *offset)
+{
+ struct drm_gem_object *obj;
+ int ret = 0;
+
+ mutex_lock(&dev->struct_mutex);
+
+ /*
+ * get offset of memory allocated for drm framebuffer.
+ * - this callback would be called by user application
+ * with DRM_IOCTL_MODE_MAP_DUMB command.
+ */
+
+ obj = drm_gem_object_lookup(dev, file_priv, handle);
+ if (!obj) {
+ DRM_ERROR("failed to lookup gem object.\n");
+ ret = -EINVAL;
+ goto unlock;
+ }
+
+ ret = drm_gem_create_mmap_offset(obj);
+ if (ret)
+ goto out;
+
+ *offset = drm_vma_node_offset_addr(&obj->vma_node);
+ DRM_DEBUG_KMS("offset = 0x%lx\n", (unsigned long)*offset);
+
+out:
+ drm_gem_object_unreference(obj);
+unlock:
+ mutex_unlock(&dev->struct_mutex);
+ return ret;
+}
+
+/*
+ * rockchip_gem_dumb_create - (struct drm_driver)->dumb_create callback
+ * function
+ *
+ * This aligns the pitch and size arguments to the minimum required. wrap
+ * this into your own function if you need bigger alignment.
+ */
+int rockchip_gem_dumb_create(struct drm_file *file_priv,
+ struct drm_device *dev,
+ struct drm_mode_create_dumb *args)
+{
+ struct rockchip_gem_object *rk_obj;
+ int min_pitch = DIV_ROUND_UP(args->width * args->bpp, 8);
+
+ if (args->pitch < min_pitch)
+ args->pitch = min_pitch;
+
+ if (args->size < args->pitch * args->height)
+ args->size = args->pitch * args->height;
+
+ rk_obj = rockchip_gem_create_with_handle(file_priv, dev, args->size,
+ &args->handle);
+
+ return PTR_ERR_OR_ZERO(rk_obj);
+}
+
+int rockchip_gem_get_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_priv)
+{
+ struct drm_rockchip_gem_info *args = data;
+ struct rockchip_gem_object *rk_obj;
+ struct drm_gem_object *obj;
+
+ mutex_lock(&dev->struct_mutex);
+
+ obj = drm_gem_object_lookup(dev, file_priv, args->handle);
+ if (!obj) {
+ DRM_ERROR("failed to lookup gem object.\n");
+ mutex_unlock(&dev->struct_mutex);
+ return -EINVAL;
+ }
+
+ rk_obj = to_rockchip_obj(obj);
+
+ args->flags = rk_obj->flags;
+ args->size = obj->size;
+
+ drm_gem_object_unreference(obj);
+ mutex_unlock(&dev->struct_mutex);
+
+ return 0;
+}
+
+int rockchip_gem_map_offset_ioctl(struct drm_device *drm, void *data,
+ struct drm_file *file_priv)
+{
+ struct drm_rockchip_gem_map_off *args = data;
+
+ return rockchip_gem_dumb_map_offset(file_priv, drm, args->handle,
+ &args->offset);
+}
+
+int rockchip_gem_create_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_priv)
+{
+ struct drm_rockchip_gem_create *args = data;
+ struct rockchip_gem_object *rk_obj;
+
+ rk_obj = rockchip_gem_create_with_handle(file_priv, dev, args->size,
+ &args->handle);
+ return PTR_ERR_OR_ZERO(rk_obj);
+}
+
+/*
+ * Allocate a sg_table for this GEM object.
+ * Note: Both the table's contents, and the sg_table itself must be freed by
+ * the caller.
+ * Returns a pointer to the newly allocated sg_table, or an ERR_PTR() error.
+ */
+struct sg_table *rockchip_gem_prime_get_sg_table(struct drm_gem_object *obj)
+{
+ struct rockchip_gem_object *rk_obj = to_rockchip_obj(obj);
+ struct drm_device *drm = obj->dev;
+ struct sg_table *sgt = NULL;
+ int ret;
+
+ sgt = kzalloc(sizeof(*sgt), GFP_KERNEL);
+ if (!sgt)
+ return ERR_PTR(-ENOMEM);
+
+ ret = dma_get_sgtable_attrs(drm->dev, sgt, rk_obj->kvaddr,
+ rk_obj->dma_addr, obj->size,
+ &rk_obj->dma_attrs);
+ if (ret) {
+ DRM_ERROR("failed to allocate sgt, %d\n", ret);
+ kfree(sgt);
+ return ERR_PTR(ret);
+ }
+
+ return sgt;
+}
+
+struct drm_gem_object *
+rockchip_gem_prime_import_sg_table(struct drm_device *dev, size_t size,
+ struct sg_table *sgt)
+{
+ struct rockchip_gem_object *rk_obj;
+
+ if (sgt->nents != 1)
+ return ERR_PTR(-EINVAL);
+
+ rk_obj = rockchip_gem_create_object(dev, size);
+ if (IS_ERR(rk_obj))
+ return ERR_PTR(-ENOMEM);
+
+ return &rk_obj->base;
+}
+
+void *rockchip_gem_prime_vmap(struct drm_gem_object *obj)
+{
+ struct rockchip_gem_object *rk_obj = to_rockchip_obj(obj);
+
+ return rk_obj->kvaddr;
+}
+
+void rockchip_gem_prime_vunmap(struct drm_gem_object *obj, void *vaddr)
+{
+ /* Nothing to do */
+}
+
+int rockchip_gem_prime_mmap(struct drm_gem_object *obj,
+ struct vm_area_struct *vma)
+{
+ struct drm_device *dev = obj->dev;
+ int ret;
+
+ mutex_lock(&dev->struct_mutex);
+ ret = drm_gem_mmap_obj(obj, obj->size, vma);
+ mutex_unlock(&dev->struct_mutex);
+
+ return ret;
+}
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_gem.h b/drivers/gpu/drm/rockchip/rockchip_drm_gem.h
new file mode 100644
index 0000000..6277dbd
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_gem.h
@@ -0,0 +1,72 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _ROCKCHIP_DRM_GEM_H
+#define _ROCKCHIP_DRM_GEM_H
+
+#define to_rockchip_obj(x) container_of(x, struct rockchip_gem_object, base)
+
+struct rockchip_gem_object {
+ struct drm_gem_object base;
+ unsigned int flags;
+
+ void *kvaddr;
+ dma_addr_t dma_addr;
+ struct dma_attrs dma_attrs;
+};
+
+struct sg_table *rockchip_gem_prime_get_sg_table(struct drm_gem_object *obj);
+struct drm_gem_object *
+rockchip_gem_prime_import_sg_table(struct drm_device *dev, size_t size,
+ struct sg_table *sgt);
+void *rockchip_gem_prime_vmap(struct drm_gem_object *obj);
+void rockchip_gem_prime_vunmap(struct drm_gem_object *obj, void *vaddr);
+int rockchip_gem_prime_mmap(struct drm_gem_object *obj,
+ struct vm_area_struct *vma);
+
+/* drm driver mmap file operations */
+int rockchip_drm_gem_mmap(struct file *filp, struct vm_area_struct *vma);
+
+/* mmap a gem object to userspace. */
+int rockchip_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma);
+
+struct rockchip_gem_object *
+ rockchip_gem_create_object(struct drm_device *drm, unsigned int size);
+
+void rockchip_gem_free_object(struct drm_gem_object *obj);
+
+int rockchip_gem_dumb_create(struct drm_file *file_priv,
+ struct drm_device *dev,
+ struct drm_mode_create_dumb *args);
+int rockchip_gem_dumb_map_offset(struct drm_file *file_priv,
+ struct drm_device *dev, uint32_t handle,
+ uint64_t *offset);
+int rockchip_gem_map_offset_ioctl(struct drm_device *drm, void *data,
+ struct drm_file *file_priv);
+/*
+ * request gem object creation and buffer allocation as the size
+ * that it is calculated with framebuffer information such as width,
+ * height and bpp.
+ */
+int rockchip_gem_create_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_priv);
+
+/* get buffer offset to map to user space. */
+int rockchip_gem_map_offset_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_priv);
+
+/* get buffer information to memory region allocated by gem. */
+int rockchip_gem_get_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_priv);
+#endif /* _ROCKCHIP_DRM_GEM_H */
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_vop.c b/drivers/gpu/drm/rockchip/rockchip_drm_vop.c
new file mode 100644
index 0000000..d2ec4d5
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_vop.c
@@ -0,0 +1,1372 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+#include <drm/drmP.h>
+#include <drm/drm_crtc.h>
+#include <drm/drm_crtc_helper.h>
+
+#include <linux/kernel.h>
+#include <linux/platform_device.h>
+#include <linux/clk.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/component.h>
+
+#include <linux/reset.h>
+#include <linux/iommu.h>
+#include <linux/delay.h>
+#include <drm/rockchip_drm.h>
+
+#include <video/of_display_timing.h>
+#include <video/of_videomode.h>
+
+#include "rockchip_drm_drv.h"
+#include "rockchip_drm_fbdev.h"
+#include "rockchip_drm_gem.h"
+#include "rockchip_drm_fb.h"
+#include "rockchip_drm_vop.h"
+
+#define VOP_DEFAULT_FRAMERATE 60
+#define VOP_MAX_WIN_SUPPORT 5
+#define VOP_DEFAULT_CURSOR 1
+#define VOP_REG(off, _mask, s) \
+ {.offset = off, \
+ .mask = _mask, \
+ .shift = s,}
+
+#define __REG_SET(x, off, mask, shift, v) \
+ vop_mask_write(x, off, (mask) << shift, (v) << shift)
+
+#define REG_SET(x, base, reg, v) \
+ __REG_SET(x, base + reg.offset, reg.mask, reg.shift, v)
+
+#define VOP_WIN_SET(x, win, name, v) \
+ REG_SET(x, win->base, win->phy->name, v)
+#define VOP_CTRL_SET(x, name, v) \
+ REG_SET(x, 0, (x)->data->ctrl->name, v)
+
+#define VOP_WIN_GET_YRGBADDR(ctx, win) \
+ vop_readl(ctx, win->base + win->phy->yrgb_mst.offset)
+
+#define to_vop_ctx(x) container_of(x, struct vop_context, crtc)
+#define to_rockchip_plane(x) container_of(x, struct rockchip_plane, base)
+
+struct rockchip_plane {
+ int id;
+ struct drm_plane base;
+ const struct vop_win *win;
+ struct vop_context *ctx;
+
+ uint32_t pending_yrgb_mst;
+ struct drm_framebuffer *front_fb;
+ struct drm_framebuffer *pending_fb;
+ bool enabled;
+};
+
+struct vop_context {
+ struct device *dev;
+ struct drm_device *drm_dev;
+ struct drm_crtc crtc;
+ struct drm_pending_vblank_event *event;
+ struct vop_driver *drv;
+ unsigned int dpms;
+ unsigned int win_mask;
+ wait_queue_head_t wait_vsync_queue;
+ atomic_t wait_vsync_event;
+
+ struct workqueue_struct *vsync_wq;
+ struct work_struct vsync_work;
+
+ /* mutex vsync_ work */
+ struct mutex vsync_mutex;
+ bool vsync_work_pending;
+
+ struct vop_driver_data *data;
+
+ uint32_t *regsbak;
+ void __iomem *regs;
+
+ /* physical map length of vop register */
+ uint32_t len;
+
+ /* one time only one process allowed to config the register */
+ spinlock_t reg_lock;
+ /* lock vop irq reg */
+ spinlock_t irq_lock;
+
+ unsigned int irq;
+
+ /* vop AHP clk */
+ struct clk *hclk;
+ /* vop dclk */
+ struct clk *dclk;
+ /* vop share memory frequency */
+ struct clk *aclk;
+ uint32_t pixclock;
+
+ int pipe;
+ bool clk_on;
+};
+
+enum vop_data_format {
+ VOP_FMT_ARGB8888 = 0,
+ VOP_FMT_RGB888,
+ VOP_FMT_RGB565,
+ VOP_FMT_YUV420SP = 4,
+ VOP_FMT_YUV422SP,
+ VOP_FMT_YUV444SP,
+};
+
+struct vop_reg_data {
+ uint32_t offset;
+ uint32_t value;
+};
+
+struct vop_reg {
+ uint32_t offset;
+ uint32_t shift;
+ uint32_t mask;
+};
+
+struct vop_ctrl {
+ struct vop_reg standby;
+ struct vop_reg gate_en;
+ struct vop_reg mmu_en;
+ struct vop_reg rgb_en;
+ struct vop_reg edp_en;
+ struct vop_reg hdmi_en;
+ struct vop_reg mipi_en;
+ struct vop_reg out_mode;
+ struct vop_reg dither_down;
+ struct vop_reg dither_up;
+ struct vop_reg pin_pol;
+
+ struct vop_reg htotal_pw;
+ struct vop_reg hact_st_end;
+ struct vop_reg vtotal_pw;
+ struct vop_reg vact_st_end;
+ struct vop_reg hpost_st_end;
+ struct vop_reg vpost_st_end;
+};
+
+struct vop_win_phy {
+ const uint32_t *data_formats;
+ uint32_t nformats;
+
+ struct vop_reg enable;
+ struct vop_reg format;
+ struct vop_reg act_info;
+ struct vop_reg dsp_info;
+ struct vop_reg dsp_st;
+ struct vop_reg yrgb_mst;
+ struct vop_reg uv_mst;
+ struct vop_reg yrgb_vir;
+ struct vop_reg uv_vir;
+
+ struct vop_reg dst_alpha_ctl;
+ struct vop_reg src_alpha_ctl;
+};
+
+struct vop_win {
+ uint32_t base;
+ const struct vop_win_phy *phy;
+};
+
+struct vop_driver_data {
+ const void *init_table;
+ int table_size;
+ const struct vop_ctrl *ctrl;
+ const struct vop_win *win[VOP_MAX_WIN_SUPPORT];
+};
+
+static const uint32_t formats_01[] = {
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_RGB888,
+ DRM_FORMAT_RGB565,
+ DRM_FORMAT_NV12,
+ DRM_FORMAT_NV16,
+ DRM_FORMAT_NV24,
+};
+
+static const uint32_t formats_234[] = {
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_RGB888,
+ DRM_FORMAT_RGB565,
+};
+
+static const struct vop_win_phy win01_data = {
+ .data_formats = formats_01,
+ .nformats = ARRAY_SIZE(formats_01),
+ .enable = VOP_REG(WIN0_CTRL0, 0x1, 0),
+ .format = VOP_REG(WIN0_CTRL0, 0x7, 1),
+ .act_info = VOP_REG(WIN0_ACT_INFO, 0x1fff1fff, 0),
+ .dsp_info = VOP_REG(WIN0_DSP_INFO, 0x1fff1fff, 0),
+ .dsp_st = VOP_REG(WIN0_DSP_ST, 0x1fff1fff, 0),
+ .yrgb_mst = VOP_REG(WIN0_YRGB_MST, 0xffffffff, 0),
+ .uv_mst = VOP_REG(WIN0_CBR_MST, 0xffffffff, 0),
+ .yrgb_vir = VOP_REG(WIN0_VIR, 0x3fff, 0),
+ .uv_vir = VOP_REG(WIN0_VIR, 0x3fff, 16),
+ .src_alpha_ctl = VOP_REG(WIN0_SRC_ALPHA_CTRL, 0xff, 0),
+ .dst_alpha_ctl = VOP_REG(WIN0_DST_ALPHA_CTRL, 0xff, 0),
+};
+
+static const struct vop_win_phy win23_data = {
+ .data_formats = formats_234,
+ .nformats = ARRAY_SIZE(formats_234),
+ .enable = VOP_REG(WIN2_CTRL0, 0x1, 0),
+ .format = VOP_REG(WIN2_CTRL0, 0x7, 1),
+ .dsp_info = VOP_REG(WIN2_DSP_INFO0, 0x0fff0fff, 0),
+ .dsp_st = VOP_REG(WIN2_DSP_ST0, 0x1fff1fff, 0),
+ .yrgb_mst = VOP_REG(WIN2_MST0, 0xffffffff, 0),
+ .yrgb_vir = VOP_REG(WIN2_VIR0_1, 0x1fff, 0),
+ .src_alpha_ctl = VOP_REG(WIN2_SRC_ALPHA_CTRL, 0xff, 0),
+ .dst_alpha_ctl = VOP_REG(WIN2_DST_ALPHA_CTRL, 0xff, 0),
+};
+
+static const struct vop_win_phy cursor_data = {
+ .data_formats = formats_234,
+ .nformats = ARRAY_SIZE(formats_234),
+ .enable = VOP_REG(HWC_CTRL0, 0x1, 0),
+ .format = VOP_REG(HWC_CTRL0, 0x7, 1),
+ .dsp_st = VOP_REG(HWC_DSP_ST, 0x1fff1fff, 0),
+ .yrgb_mst = VOP_REG(HWC_MST, 0xffffffff, 0),
+};
+
+static const struct vop_win win0 = {
+ .base = 0,
+ .phy = &win01_data,
+};
+
+static const struct vop_win win1 = {
+ .base = 0x40,
+ .phy = &win01_data,
+};
+
+static const struct vop_win win2 = {
+ .base = 0,
+ .phy = &win23_data,
+};
+
+static const struct vop_win win3 = {
+ .base = 0x50,
+ .phy = &win23_data,
+};
+
+static const struct vop_win win_cursor = {
+ .base = 0,
+ .phy = &cursor_data,
+};
+
+static const struct vop_ctrl ctrl_data = {
+ .standby = VOP_REG(SYS_CTRL, 0x1, 22),
+ .gate_en = VOP_REG(SYS_CTRL, 0x1, 23),
+ .mmu_en = VOP_REG(SYS_CTRL, 0x1, 20),
+ .rgb_en = VOP_REG(SYS_CTRL, 0x1, 12),
+ .hdmi_en = VOP_REG(SYS_CTRL, 0x1, 13),
+ .edp_en = VOP_REG(SYS_CTRL, 0x1, 14),
+ .mipi_en = VOP_REG(SYS_CTRL, 0x1, 15),
+ .dither_down = VOP_REG(DSP_CTRL1, 0xf, 1),
+ .dither_up = VOP_REG(DSP_CTRL1, 0x1, 6),
+ .out_mode = VOP_REG(DSP_CTRL0, 0xf, 0),
+ .pin_pol = VOP_REG(DSP_CTRL0, 0xf, 4),
+ .htotal_pw = VOP_REG(DSP_HTOTAL_HS_END, 0x1fff1fff, 0),
+ .hact_st_end = VOP_REG(DSP_HACT_ST_END, 0x1fff1fff, 0),
+ .vtotal_pw = VOP_REG(DSP_VTOTAL_VS_END, 0x1fff1fff, 0),
+ .vact_st_end = VOP_REG(DSP_VACT_ST_END, 0x1fff1fff, 0),
+ .hpost_st_end = VOP_REG(POST_DSP_HACT_INFO, 0x1fff1fff, 0),
+ .vpost_st_end = VOP_REG(POST_DSP_VACT_INFO, 0x1fff1fff, 0),
+};
+
+static const struct vop_reg_data vop_init_reg_table[] = {
+ {SYS_CTRL, 0x00801000},
+ {DSP_CTRL0, 0x00000000},
+ {WIN0_CTRL0, 0x00000080},
+ {WIN1_CTRL0, 0x00000080},
+};
+
+static const struct vop_driver_data rockchip_rk3288_vop = {
+ .init_table = vop_init_reg_table,
+ .table_size = ARRAY_SIZE(vop_init_reg_table),
+ .ctrl = &ctrl_data,
+ .win[0] = &win0,
+ .win[1] = &win1,
+ .win[2] = &win2,
+ .win[3] = &win3,
+ .win[4] = &win_cursor,
+};
+
+static const struct of_device_id vop_driver_dt_match[] = {
+ { .compatible = "rockchip,rk3288-vop",
+ .data = (void *)&rockchip_rk3288_vop },
+ {},
+};
+
+static inline void vop_writel(struct vop_context *ctx,
+ uint32_t offset, uint32_t v)
+{
+ writel(v, ctx->regs + offset);
+ ctx->regsbak[offset >> 2] = v;
+}
+
+static inline uint32_t vop_readl(struct vop_context *ctx, uint32_t offset)
+{
+ return readl(ctx->regs + offset);
+}
+
+static inline void vop_cfg_done(struct vop_context *ctx)
+{
+ writel(0x01, ctx->regs + REG_CFG_DONE);
+}
+
+static inline void vop_mask_write(struct vop_context *ctx,
+ uint32_t offset, uint32_t mask, uint32_t v)
+{
+ if (mask) {
+ uint32_t cached_val = ctx->regsbak[offset >> 2];
+
+ cached_val = (cached_val & ~mask) | v;
+ writel(cached_val, ctx->regs + offset);
+ ctx->regsbak[offset >> 2] = cached_val;
+ }
+}
+
+static inline struct vop_driver_data *vop_get_driver_data(struct device *dev)
+{
+ const struct of_device_id *of_id =
+ of_match_device(vop_driver_dt_match, dev);
+
+ return (struct vop_driver_data *)of_id->data;
+}
+
+static enum vop_data_format vop_convert_format(uint32_t format)
+{
+ switch (format) {
+ case DRM_FORMAT_XRGB8888:
+ case DRM_FORMAT_ARGB8888:
+ return VOP_FMT_ARGB8888;
+ case DRM_FORMAT_RGB888:
+ return VOP_FMT_RGB888;
+ case DRM_FORMAT_RGB565:
+ return VOP_FMT_RGB565;
+ case DRM_FORMAT_NV12:
+ return VOP_FMT_YUV420SP;
+ case DRM_FORMAT_NV16:
+ return VOP_FMT_YUV422SP;
+ case DRM_FORMAT_NV24:
+ return VOP_FMT_YUV444SP;
+ default:
+ DRM_ERROR("unsupport format[%08x]\n", format);
+ return -EINVAL;
+ }
+}
+
+static bool is_alpha_support(uint32_t format)
+{
+ switch (format) {
+ case DRM_FORMAT_ARGB8888:
+ return true;
+ default:
+ return false;
+ }
+}
+
+/* TODO(djkurtz): move generic 'setup slave rk_iommu' code somewhere common */
+int vop_iommu_init(struct vop_context *ctx)
+{
+ struct device *dev = ctx->dev;
+ struct device_node *np = dev->of_node;
+ struct platform_device *pd;
+ int count;
+ int ret;
+ struct of_phandle_args args;
+
+ /* Each VOP must have exactly one iommu node, with no args */
+ count = of_count_phandle_with_args(np, "iommus", "#iommu-cells");
+ if (count != 1) {
+ dev_err(dev, "of_count_phandle_with_args(%s) => %d\n",
+ np->full_name, count);
+ return -EINVAL;
+ }
+
+ ret = of_parse_phandle_with_args(np, "iommus", "#iommu-cells", 0,
+ &args);
+ if (ret) {
+ dev_err(dev, "of_parse_phandle_with_args(%s) => %d\n",
+ np->full_name, ret);
+ return ret;
+ }
+ if (args.args_count != 0) {
+ dev_err(dev, "incorrect number of iommu params found for %s (found %d, expected 0)\n",
+ args.np->full_name, args.args_count);
+ return -EINVAL;
+ }
+
+ pd = of_find_device_by_node(args.np);
+ of_node_put(args.np);
+ if (!pd) {
+ dev_err(dev, "iommu %s not found\n", args.np->full_name);
+ return -EPROBE_DEFER;
+ }
+
+ /* TODO(djkurtz): handle multiple slave iommus for a single master */
+ dev->archdata.iommu = &pd->dev;
+
+ ret = rockchip_drm_dma_attach_device(ctx->drm_dev, dev);
+ if (ret) {
+ dev_err(dev, "failed to attach to drm dma mapping, %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void vop_iommu_fini(struct vop_context *ctx)
+{
+ rockchip_drm_dma_detach_device(ctx->drm_dev, ctx->dev);
+}
+
+static int rockchip_plane_get_size(int start, unsigned length, unsigned last)
+{
+ int end = start + length;
+ int size = 0;
+
+ if (start <= 0) {
+ if (end > 0)
+ size = min_t(unsigned, end, last);
+ } else if (start <= last) {
+ size = min_t(unsigned, last - start, length);
+ }
+
+ return size;
+}
+
+static int vop_clk_enable(struct vop_context *ctx)
+{
+ int ret;
+
+ if (!ctx->clk_on) {
+ ret = clk_prepare_enable(ctx->hclk);
+ if (ret < 0) {
+ dev_err(ctx->dev, "failed to enable hclk\n");
+ return ret;
+ }
+
+ ret = clk_prepare_enable(ctx->dclk);
+ if (ret < 0) {
+ dev_err(ctx->dev, "failed to enable dclk\n");
+ goto err_dclk;
+ }
+
+ ret = clk_prepare_enable(ctx->aclk);
+ if (ret < 0) {
+ dev_err(ctx->dev, "failed to enable aclk\n");
+ goto err_aclk;
+ }
+ ctx->clk_on = true;
+ }
+
+ return ret;
+err_aclk:
+ clk_disable_unprepare(ctx->aclk);
+err_dclk:
+ clk_disable_unprepare(ctx->hclk);
+ return ret;
+}
+
+static void vop_clk_disable(struct vop_context *ctx)
+{
+ if (ctx->clk_on) {
+ clk_disable_unprepare(ctx->dclk);
+ clk_disable_unprepare(ctx->hclk);
+ clk_disable_unprepare(ctx->aclk);
+ ctx->clk_on = false;
+ }
+}
+
+static void vop_power_on(struct vop_context *ctx)
+{
+ if (vop_clk_enable(ctx) < 0) {
+ dev_err(ctx->dev, "failed to enable clks\n");
+ return;
+ }
+
+ spin_lock(&ctx->reg_lock);
+
+ VOP_CTRL_SET(ctx, standby, 0);
+
+ spin_unlock(&ctx->reg_lock);
+}
+
+static void vop_power_off(struct vop_context *ctx)
+{
+ spin_lock(&ctx->reg_lock);
+
+ VOP_CTRL_SET(ctx, standby, 1);
+
+ spin_unlock(&ctx->reg_lock);
+
+ vop_clk_disable(ctx);
+}
+
+static int rockchip_update_plane(struct drm_plane *plane, struct drm_crtc *crtc,
+ struct drm_framebuffer *fb, int crtc_x,
+ int crtc_y, unsigned int crtc_w,
+ unsigned int crtc_h, uint32_t src_x,
+ uint32_t src_y, uint32_t src_w, uint32_t src_h)
+{
+ struct rockchip_plane *rockchip_plane = to_rockchip_plane(plane);
+ const struct vop_win *win = rockchip_plane->win;
+ struct vop_context *ctx = to_vop_ctx(crtc);
+ struct drm_gem_object *obj;
+ struct rockchip_gem_object *rk_obj;
+ unsigned long offset;
+ unsigned int actual_w;
+ unsigned int actual_h;
+ unsigned int dsp_stx;
+ unsigned int dsp_sty;
+ unsigned int y_vir_stride;
+ dma_addr_t yrgb_mst;
+ enum vop_data_format format;
+ uint32_t val;
+ bool is_alpha;
+
+ if (!win) {
+ DRM_ERROR("can't find win data for vop, failed\n");
+ return -EINVAL;
+ }
+
+ obj = rockchip_fb_get_gem_obj(fb, 0);
+ if (!obj) {
+ DRM_ERROR("fail to get rockchip gem object from framebuffer\n");
+ return -EINVAL;
+ }
+
+ rk_obj = to_rockchip_obj(obj);
+
+ yrgb_mst = rk_obj->dma_addr;
+ if (yrgb_mst <= 0)
+ return -ENOMEM;
+
+ actual_w = rockchip_plane_get_size(crtc_x,
+ crtc_w, crtc->mode.hdisplay);
+ actual_h = rockchip_plane_get_size(crtc_y,
+ crtc_h, crtc->mode.vdisplay);
+ if (crtc_x < 0) {
+ if (actual_w)
+ src_x -= crtc_x;
+ crtc_x = 0;
+ }
+
+ if (crtc_y < 0) {
+ if (actual_h)
+ src_y -= crtc_y;
+ crtc_y = 0;
+ }
+
+ dsp_stx = crtc_x + crtc->mode.htotal - crtc->mode.hsync_start;
+ dsp_sty = crtc_y + crtc->mode.vtotal - crtc->mode.vsync_start;
+
+ offset = src_x * (fb->bits_per_pixel >> 3);
+ offset += src_y * fb->pitches[0];
+
+ y_vir_stride = fb->pitches[0] / (fb->bits_per_pixel >> 3);
+ is_alpha = is_alpha_support(fb->pixel_format);
+ format = vop_convert_format(fb->pixel_format);
+
+ spin_lock(&ctx->reg_lock);
+
+ VOP_WIN_SET(ctx, win, format, format);
+ VOP_WIN_SET(ctx, win, yrgb_vir, y_vir_stride);
+ yrgb_mst += offset;
+ VOP_WIN_SET(ctx, win, yrgb_mst, yrgb_mst);
+ VOP_WIN_SET(ctx, win, act_info,
+ ((actual_h - 1) << 16) | (actual_w - 1));
+ VOP_WIN_SET(ctx, win, dsp_info,
+ ((actual_h - 1) << 16) | (actual_w - 1));
+ VOP_WIN_SET(ctx, win, dsp_st, (dsp_sty << 16) | dsp_stx);
+ if (is_alpha) {
+ VOP_WIN_SET(ctx, win, dst_alpha_ctl,
+ DST_FACTOR_M0(ALPHA_SRC_INVERSE));
+ val = SRC_ALPHA_EN(1) | SRC_COLOR_M0(ALPHA_SRC_PRE_MUL) |
+ SRC_ALPHA_M0(ALPHA_STRAIGHT) |
+ SRC_BLEND_M0(ALPHA_PER_PIX) |
+ SRC_ALPHA_CAL_M0(ALPHA_NO_SATURATION) |
+ SRC_FACTOR_M0(ALPHA_ONE);
+ VOP_WIN_SET(ctx, win, src_alpha_ctl, val);
+ } else {
+ VOP_WIN_SET(ctx, win, src_alpha_ctl, SRC_ALPHA_EN(0));
+ }
+
+ VOP_WIN_SET(ctx, win, enable, 1);
+
+ spin_unlock(&ctx->reg_lock);
+
+ mutex_lock(&ctx->vsync_mutex);
+
+ /*
+ * Because the buffer set to vop take effect at frame start time,
+ * we need make sure old buffer is not in use before we release
+ * it.
+ * reference the framebuffer, and unference it when it swap out of vop.
+ */
+ if (fb != rockchip_plane->front_fb) {
+ drm_framebuffer_reference(fb);
+ rockchip_plane->pending_fb = fb;
+ rockchip_plane->pending_yrgb_mst = yrgb_mst;
+ ctx->vsync_work_pending = true;
+ }
+ rockchip_plane->enabled = true;
+
+ mutex_unlock(&ctx->vsync_mutex);
+
+ spin_lock(&ctx->reg_lock);
+ vop_cfg_done(ctx);
+ spin_unlock(&ctx->reg_lock);
+
+ return 0;
+}
+
+static int rockchip_disable_plane(struct drm_plane *plane)
+{
+ struct rockchip_plane *rockchip_plane = to_rockchip_plane(plane);
+ struct vop_context *ctx = rockchip_plane->ctx;
+ const struct vop_win *win = rockchip_plane->win;
+
+ spin_lock(&ctx->reg_lock);
+
+ VOP_WIN_SET(ctx, win, enable, 0);
+ vop_cfg_done(ctx);
+
+ spin_unlock(&ctx->reg_lock);
+
+ mutex_lock(&ctx->vsync_mutex);
+
+ /*
+ * clear the pending framebuffer and set vsync_work_pending true,
+ * so that the framebuffer will unref at the next vblank.
+ */
+ if (rockchip_plane->pending_fb) {
+ drm_framebuffer_unreference(rockchip_plane->pending_fb);
+ rockchip_plane->pending_fb = NULL;
+ }
+
+ rockchip_plane->enabled = false;
+ ctx->vsync_work_pending = true;
+
+ mutex_unlock(&ctx->vsync_mutex);
+
+ return 0;
+}
+
+static void rockchip_plane_destroy(struct drm_plane *plane)
+{
+ struct rockchip_plane *rockchip_plane = to_rockchip_plane(plane);
+ struct vop_context *ctx = rockchip_plane->ctx;
+
+ rockchip_disable_plane(plane);
+ drm_plane_cleanup(plane);
+ ctx->win_mask &= ~(1 << rockchip_plane->id);
+ kfree(rockchip_plane);
+}
+
+static const struct drm_plane_funcs rockchip_plane_funcs = {
+ .update_plane = rockchip_update_plane,
+ .disable_plane = rockchip_disable_plane,
+ .destroy = rockchip_plane_destroy,
+};
+
+struct drm_plane *rockchip_plane_init(struct vop_context *ctx,
+ unsigned long possible_crtcs,
+ enum drm_plane_type type)
+{
+ struct rockchip_plane *rockchip_plane;
+ struct vop_driver_data *vop_data = ctx->data;
+ const struct vop_win *win;
+ int i;
+ int err;
+
+ rockchip_plane = kzalloc(sizeof(*rockchip_plane), GFP_KERNEL);
+ if (!rockchip_plane)
+ return NULL;
+
+ for (i = 0; i < VOP_MAX_WIN_SUPPORT; i++) {
+ if (!(ctx->win_mask & (1 << i))) {
+ win = vop_data->win[i];
+ break;
+ }
+ }
+
+ if (VOP_MAX_WIN_SUPPORT == i) {
+ DRM_ERROR("failed to find win\n");
+ kfree(rockchip_plane);
+ return NULL;
+ }
+
+ ctx->win_mask |= (1 << i);
+ rockchip_plane->id = i;
+ rockchip_plane->win = win;
+ rockchip_plane->ctx = ctx;
+
+ err = drm_universal_plane_init(ctx->drm_dev, &rockchip_plane->base,
+ possible_crtcs, &rockchip_plane_funcs,
+ win->phy->data_formats,
+ win->phy->nformats, type);
+ if (err) {
+ DRM_ERROR("failed to initialize plane\n");
+ kfree(rockchip_plane);
+ return NULL;
+ }
+
+ return &rockchip_plane->base;
+}
+
+int rockchip_drm_crtc_enable_vblank(struct drm_device *dev, int pipe)
+{
+ struct vop_context *ctx = to_vop_ctx(rockchip_find_crtc(dev, pipe));
+ unsigned long flags;
+
+ if (ctx->dpms != DRM_MODE_DPMS_ON)
+ return -EPERM;
+
+ spin_lock_irqsave(&ctx->irq_lock, flags);
+
+ vop_mask_write(ctx, INTR_CTRL0, LINE_FLAG_INTR_MASK,
+ LINE_FLAG_INTR_EN(1));
+
+ spin_unlock_irqrestore(&ctx->irq_lock, flags);
+
+ return 0;
+}
+
+void rockchip_drm_crtc_disable_vblank(struct drm_device *dev, int pipe)
+{
+ struct vop_context *ctx = to_vop_ctx(rockchip_find_crtc(dev, pipe));
+ unsigned long flags;
+
+ if (ctx->dpms != DRM_MODE_DPMS_ON)
+ return;
+ spin_lock_irqsave(&ctx->irq_lock, flags);
+ vop_mask_write(ctx, INTR_CTRL0, LINE_FLAG_INTR_MASK,
+ LINE_FLAG_INTR_EN(0));
+ spin_unlock_irqrestore(&ctx->irq_lock, flags);
+}
+
+static void rockchip_drm_crtc_dpms(struct drm_crtc *crtc, int mode)
+{
+ struct vop_context *ctx = to_vop_ctx(crtc);
+
+ DRM_DEBUG_KMS("crtc[%d] mode[%d]\n", crtc->base.id, mode);
+
+ if (ctx->dpms == mode) {
+ DRM_DEBUG_KMS("desired dpms mode is same as previous one.\n");
+ return;
+ }
+ if (mode > DRM_MODE_DPMS_ON) {
+ /* wait for the completion of page flip. */
+ if (!wait_event_timeout(ctx->wait_vsync_queue,
+ !atomic_read(&ctx->wait_vsync_event),
+ HZ/20))
+ DRM_DEBUG_KMS("vblank wait timed out.\n");
+ drm_vblank_off(crtc->dev, ctx->pipe);
+ }
+
+ switch (mode) {
+ case DRM_MODE_DPMS_ON:
+ vop_power_on(ctx);
+ break;
+ case DRM_MODE_DPMS_STANDBY:
+ case DRM_MODE_DPMS_SUSPEND:
+ case DRM_MODE_DPMS_OFF:
+ vop_power_off(ctx);
+ break;
+ default:
+ DRM_DEBUG_KMS("unspecified mode %d\n", mode);
+ break;
+ }
+
+ ctx->dpms = mode;
+}
+
+static void rockchip_drm_crtc_prepare(struct drm_crtc *crtc)
+{
+ rockchip_drm_crtc_dpms(crtc, DRM_MODE_DPMS_ON);
+}
+
+static bool rockchip_drm_crtc_mode_fixup(struct drm_crtc *crtc,
+ const struct drm_display_mode *mode,
+ struct drm_display_mode *adjusted_mode)
+{
+ /* just do dummy now */
+
+ return true;
+}
+
+static int rockchip_drm_crtc_mode_set_base(struct drm_crtc *crtc, int x, int y,
+ struct drm_framebuffer *old_fb);
+
+static int rockchip_drm_crtc_mode_set(struct drm_crtc *crtc,
+ struct drm_display_mode *mode,
+ struct drm_display_mode *adjusted_mode,
+ int x, int y,
+ struct drm_framebuffer *fb)
+{
+ struct vop_context *ctx = to_vop_ctx(crtc);
+ u16 hsync_len = adjusted_mode->hsync_end - adjusted_mode->hsync_start;
+ u16 left_margin = adjusted_mode->htotal - adjusted_mode->hsync_end;
+ u16 vsync_len = adjusted_mode->vsync_end - adjusted_mode->vsync_start;
+ u16 upper_margin = adjusted_mode->vtotal - adjusted_mode->vsync_end;
+ u16 hdisplay = adjusted_mode->hdisplay;
+ u16 vdisplay = adjusted_mode->vdisplay;
+ u16 htotal = adjusted_mode->htotal;
+ u16 vtotal = adjusted_mode->vtotal;
+ struct rockchip_display_mode *priv_mode =
+ (void *)adjusted_mode->private;
+ unsigned long flags;
+ int ret;
+ uint32_t val;
+
+ /* nothing to do if we haven't set the mode yet */
+ if (adjusted_mode->htotal == 0 || adjusted_mode->vtotal == 0)
+ return -EINVAL;
+
+ if (!priv_mode) {
+ DRM_ERROR("fail to found display output type[%d]\n",
+ priv_mode->out_type);
+ return -EINVAL;
+ }
+
+ ret = rockchip_drm_crtc_mode_set_base(crtc, x, y, fb);
+ if (ret)
+ return ret;
+
+ switch (priv_mode->out_type) {
+ case ROCKCHIP_DISPLAY_TYPE_RGB:
+ case ROCKCHIP_DISPLAY_TYPE_LVDS:
+ VOP_CTRL_SET(ctx, rgb_en, 1);
+ VOP_CTRL_SET(ctx, out_mode, ROCKCHIP_OUTFACE_P888);
+ break;
+ case ROCKCHIP_DISPLAY_TYPE_EDP:
+ VOP_CTRL_SET(ctx, edp_en, 1);
+ VOP_CTRL_SET(ctx, out_mode, ROCKCHIP_OUTFACE_AAAA);
+ break;
+ case ROCKCHIP_DISPLAY_TYPE_HDMI:
+ VOP_CTRL_SET(ctx, out_mode, ROCKCHIP_OUTFACE_AAAA);
+ VOP_CTRL_SET(ctx, hdmi_en, 1);
+ break;
+ default:
+ DRM_ERROR("unsupport out type[%d]\n", priv_mode->out_type);
+ return -EINVAL;
+ };
+
+ val = 0x8;
+ val |= (adjusted_mode->flags & DRM_MODE_FLAG_NHSYNC) ? 1 : 0;
+ val |= (adjusted_mode->flags & DRM_MODE_FLAG_NVSYNC) ? (1 << 1) : 0;
+ VOP_CTRL_SET(ctx, pin_pol, val);
+
+ VOP_CTRL_SET(ctx, htotal_pw, (htotal << 16) | hsync_len);
+ val = (hsync_len + left_margin) << 16;
+ val |= hsync_len + left_margin + hdisplay;
+ VOP_CTRL_SET(ctx, hact_st_end, val);
+ VOP_CTRL_SET(ctx, hpost_st_end, val);
+
+ VOP_CTRL_SET(ctx, vtotal_pw, (vtotal << 16) | vsync_len);
+ val = (vsync_len + upper_margin) << 16;
+ val |= vsync_len + upper_margin + vdisplay;
+ VOP_CTRL_SET(ctx, vact_st_end, val);
+ VOP_CTRL_SET(ctx, vpost_st_end, val);
+
+ spin_lock_irqsave(&ctx->irq_lock, flags);
+
+ vop_mask_write(ctx, INTR_CTRL0, DSP_LINE_NUM_MASK,
+ DSP_LINE_NUM(vsync_len + upper_margin + vdisplay));
+
+ spin_unlock_irqrestore(&ctx->irq_lock, flags);
+
+ clk_set_rate(ctx->dclk, adjusted_mode->clock * 1000);
+
+ return 0;
+}
+
+static int rockchip_drm_crtc_mode_set_base(struct drm_crtc *crtc, int x, int y,
+ struct drm_framebuffer *old_fb)
+{
+ unsigned int crtc_w;
+ unsigned int crtc_h;
+ int ret;
+
+ crtc_w = crtc->primary->fb->width - crtc->x;
+ crtc_h = crtc->primary->fb->height - crtc->y;
+
+ ret = rockchip_update_plane(crtc->primary, crtc, crtc->primary->fb, 0,
+ 0, crtc_w, crtc_h, crtc->x, crtc->y, crtc_w,
+ crtc_h);
+ if (ret < 0) {
+ DRM_ERROR("fail to update plane\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void rockchip_drm_crtc_commit(struct drm_crtc *crtc)
+{
+ /* just do dummy now */
+}
+
+static const struct drm_crtc_helper_funcs rockchip_crtc_helper_funcs = {
+ .dpms = rockchip_drm_crtc_dpms,
+ .prepare = rockchip_drm_crtc_prepare,
+ .mode_fixup = rockchip_drm_crtc_mode_fixup,
+ .mode_set = rockchip_drm_crtc_mode_set,
+ .mode_set_base = rockchip_drm_crtc_mode_set_base,
+ .commit = rockchip_drm_crtc_commit,
+};
+
+static int rockchip_drm_crtc_page_flip(struct drm_crtc *crtc,
+ struct drm_framebuffer *fb,
+ struct drm_pending_vblank_event *event,
+ uint32_t page_flip_flags)
+{
+ struct drm_device *dev = crtc->dev;
+ struct vop_context *ctx = to_vop_ctx(crtc);
+ struct drm_framebuffer *old_fb = crtc->primary->fb;
+ unsigned int crtc_w;
+ unsigned int crtc_h;
+ int ret;
+
+ /* when the page flip is requested, crtc's dpms should be on */
+ if (ctx->dpms > DRM_MODE_DPMS_ON) {
+ DRM_DEBUG("failed page flip request at dpms[%d].\n", ctx->dpms);
+ return 0;
+ }
+
+ ret = drm_vblank_get(dev, ctx->pipe);
+ if (ret) {
+ DRM_DEBUG("failed to acquire vblank counter\n");
+ return ret;
+ }
+
+ spin_lock_irq(&dev->event_lock);
+ if (ctx->event) {
+ spin_unlock_irq(&dev->event_lock);
+ DRM_ERROR("already pending flip!\n");
+ return -EBUSY;
+ }
+ ctx->event = event;
+ atomic_set(&ctx->wait_vsync_event, 1);
+ spin_unlock_irq(&dev->event_lock);
+
+ crtc->primary->fb = fb;
+ crtc_w = crtc->primary->fb->width - crtc->x;
+ crtc_h = crtc->primary->fb->height - crtc->y;
+
+ ret = rockchip_update_plane(crtc->primary, crtc, fb, 0, 0, crtc_w,
+ crtc_h, crtc->x, crtc->y, crtc_w, crtc_h);
+ if (ret) {
+ crtc->primary->fb = old_fb;
+
+ spin_lock_irq(&dev->event_lock);
+ drm_vblank_put(dev, ctx->pipe);
+ atomic_set(&ctx->wait_vsync_event, 0);
+ ctx->event = NULL;
+ spin_unlock_irq(&dev->event_lock);
+ }
+
+ return ret;
+}
+
+void rockchip_drm_crtc_finish_pageflip(struct drm_device *dev, int pipe)
+{
+ struct rockchip_drm_private *dev_priv = dev->dev_private;
+ struct drm_crtc *drm_crtc = dev_priv->crtc[pipe];
+ struct vop_context *ctx;
+ unsigned long flags;
+
+ if (!drm_crtc)
+ return;
+
+ ctx = to_vop_ctx(drm_crtc);
+
+ spin_lock_irqsave(&dev->event_lock, flags);
+
+ if (ctx->event) {
+ drm_send_vblank_event(dev, -1, ctx->event);
+ drm_vblank_put(dev, pipe);
+ atomic_set(&ctx->wait_vsync_event, 0);
+ wake_up(&ctx->wait_vsync_queue);
+ ctx->event = NULL;
+ }
+
+ spin_unlock_irqrestore(&dev->event_lock, flags);
+}
+
+void rockchip_drm_crtc_cancel_pending_flip(struct drm_device *dev)
+{
+ int i;
+
+ for (i = 0; i < dev->num_crtcs; i++)
+ rockchip_drm_crtc_finish_pageflip(dev, i);
+}
+
+static void rockchip_drm_crtc_destroy(struct drm_crtc *crtc)
+{
+ struct vop_context *ctx = to_vop_ctx(crtc);
+ struct rockchip_drm_private *private = crtc->dev->dev_private;
+
+ private->crtc[ctx->pipe] = NULL;
+ drm_crtc_cleanup(crtc);
+}
+
+static const struct drm_crtc_funcs rockchip_crtc_funcs = {
+ .set_config = drm_crtc_helper_set_config,
+ .page_flip = rockchip_drm_crtc_page_flip,
+ .destroy = rockchip_drm_crtc_destroy,
+};
+
+static void rockchip_vsync_worker(struct work_struct *work)
+{
+ struct vop_context *ctx = container_of(work, struct vop_context,
+ vsync_work);
+ struct drm_device *drm = ctx->drm_dev;
+ struct rockchip_drm_private *dev_priv = drm->dev_private;
+ struct drm_crtc *crtc = dev_priv->crtc[ctx->pipe];
+ struct rockchip_plane *rockchip_plane;
+ struct drm_plane *plane;
+ uint32_t yrgb_mst;
+
+ mutex_lock(&ctx->vsync_mutex);
+
+ ctx->vsync_work_pending = false;
+
+ list_for_each_entry(plane, &drm->mode_config.plane_list, head) {
+ rockchip_plane = to_rockchip_plane(plane);
+
+ if (rockchip_plane->ctx != ctx)
+ continue;
+ if (rockchip_plane->enabled && !rockchip_plane->pending_fb)
+ continue;
+ if (!rockchip_plane->enabled && !rockchip_plane->front_fb)
+ continue;
+ /*
+ * make sure the yrgb_mst take effect, so that
+ * we can unreference the old framebuffer.
+ */
+ yrgb_mst = VOP_WIN_GET_YRGBADDR(ctx, rockchip_plane->win);
+ if (rockchip_plane->pending_yrgb_mst != yrgb_mst) {
+ /*
+ * some plane no complete, unref at next vblank
+ */
+ ctx->vsync_work_pending = true;
+ continue;
+ }
+
+ /*
+ * drm_framebuffer_unreference maybe call iommu unmap,
+ * and iommu not allow unmap buffer at irq context,
+ * so we do drm_framebuffer_unreference at queue_work.
+ */
+ if (rockchip_plane->front_fb)
+ drm_framebuffer_unreference(rockchip_plane->front_fb);
+
+ rockchip_plane->front_fb = rockchip_plane->pending_fb;
+ rockchip_plane->pending_fb = NULL;
+
+ /*
+ * if primary plane flip complete, sending the event to
+ * userspace
+ */
+ if (&rockchip_plane->base == crtc->primary)
+ rockchip_drm_crtc_finish_pageflip(ctx->drm_dev,
+ ctx->pipe);
+ }
+
+ mutex_unlock(&ctx->vsync_mutex);
+}
+
+static irqreturn_t rockchip_vop_isr(int irq, void *data)
+{
+ struct vop_context *ctx = data;
+ uint32_t intr0_reg;
+ unsigned long flags;
+
+ intr0_reg = vop_readl(ctx, INTR_CTRL0);
+ if (intr0_reg & LINE_FLAG_INTR) {
+ spin_lock_irqsave(&ctx->irq_lock, flags);
+ vop_writel(ctx, INTR_CTRL0, intr0_reg | LINE_FLAG_INTR_CLR);
+ spin_unlock_irqrestore(&ctx->irq_lock, flags);
+ } else {
+ return IRQ_NONE;
+ }
+
+ drm_handle_vblank(ctx->drm_dev, ctx->pipe);
+ if (ctx->vsync_work_pending)
+ queue_work(ctx->vsync_wq, &ctx->vsync_work);
+
+ return IRQ_HANDLED;
+}
+
+static int vop_create_crtc(struct vop_context *ctx)
+{
+ struct device *dev = ctx->dev;
+ struct drm_device *drm_dev = ctx->drm_dev;
+ struct drm_plane *primary, *cursor;
+ unsigned long possible_crtcs;
+ struct drm_crtc *crtc;
+ int ret;
+ int nr;
+
+ ctx->win_mask = 0;
+ crtc = &ctx->crtc;
+
+ ret = rockchip_drm_add_crtc(drm_dev, crtc, dev->of_node);
+ if (ret < 0)
+ return ret;
+ ctx->pipe = ret;
+
+ possible_crtcs = (1 << ctx->pipe);
+
+ primary = rockchip_plane_init(ctx, possible_crtcs,
+ DRM_PLANE_TYPE_PRIMARY);
+ if (!primary) {
+ DRM_ERROR("fail to init primary plane\n");
+ return -EINVAL;
+ }
+
+ for (nr = 1; nr < ROCKCHIP_MAX_PLANE; nr++) {
+ if (nr == VOP_DEFAULT_CURSOR) {
+ cursor = rockchip_plane_init(ctx, possible_crtcs,
+ DRM_PLANE_TYPE_CURSOR);
+ if (!cursor) {
+ DRM_ERROR("fail to init cursor plane\n");
+ return -EINVAL;
+ }
+ } else {
+ struct drm_plane *plane;
+
+ plane = rockchip_plane_init(ctx, possible_crtcs,
+ DRM_PLANE_TYPE_OVERLAY);
+ if (!plane) {
+ DRM_ERROR("fail to init overlay plane\n");
+ return -EINVAL;
+ }
+ }
+ }
+
+ drm_crtc_init_with_planes(drm_dev, crtc, primary, cursor,
+ &rockchip_crtc_funcs);
+ drm_crtc_helper_add(crtc, &rockchip_crtc_helper_funcs);
+
+ return 0;
+}
+
+static int rockchip_vop_initial(struct vop_context *ctx)
+{
+ struct vop_driver_data *vop_data = ctx->data;
+ const struct vop_reg_data *init_table = vop_data->init_table;
+ struct reset_control *rst;
+ int i, ret;
+
+ ctx->hclk = devm_clk_get(ctx->dev, "hclk_vop");
+ if (IS_ERR(ctx->hclk)) {
+ dev_err(ctx->dev, "failed to get hclk source\n");
+ return PTR_ERR(ctx->hclk);
+ }
+ ctx->aclk = devm_clk_get(ctx->dev, "aclk_vop");
+ if (IS_ERR(ctx->aclk)) {
+ dev_err(ctx->dev, "failed to get aclk source\n");
+ return PTR_ERR(ctx->aclk);
+ }
+ ctx->dclk = devm_clk_get(ctx->dev, "dclk_vop");
+ if (IS_ERR(ctx->dclk)) {
+ dev_err(ctx->dev, "failed to get dclk source\n");
+ return PTR_ERR(ctx->dclk);
+ }
+
+ ret = vop_clk_enable(ctx);
+ if (ret < 0)
+ return ret;
+
+ /*
+ * do hclk_reset, reset all vop registers.
+ */
+ rst = devm_reset_control_get(ctx->dev, "ahb");
+ if (IS_ERR(rst)) {
+ dev_err(ctx->dev, "failed to get ahb reset\n");
+ return PTR_ERR(rst);
+ }
+ reset_control_assert(rst);
+ usleep_range(10, 20);
+ reset_control_deassert(rst);
+
+ memcpy(ctx->regsbak, ctx->regs, ctx->len);
+
+ for (i = 0; i < vop_data->table_size; i++)
+ vop_writel(ctx, init_table[i].offset, init_table[i].value);
+
+ for (i = 0; i < VOP_MAX_WIN_SUPPORT; i++)
+ VOP_WIN_SET(ctx, vop_data->win[i], enable, 0);
+
+ vop_cfg_done(ctx);
+
+ /*
+ * do dclk_reset, let all win config take affect, and then we can enable
+ * iommu safe.
+ */
+ rst = devm_reset_control_get(ctx->dev, "dclk");
+ if (IS_ERR(rst)) {
+ dev_err(ctx->dev, "failed to get dclk reset\n");
+ return PTR_ERR(rst);
+ }
+ reset_control_assert(rst);
+ usleep_range(10, 20);
+ reset_control_deassert(rst);
+
+ ctx->dpms = DRM_MODE_DPMS_ON;
+
+ return 0;
+}
+
+static int vop_bind(struct device *dev, struct device *master, void *data)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct vop_driver_data *vop_data = vop_get_driver_data(dev);
+ struct drm_device *drm_dev = data;
+ struct vop_context *ctx;
+ struct resource *res;
+ int ret;
+
+ if (!vop_data)
+ return -ENODEV;
+
+ ctx = devm_kzalloc(dev, sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return -ENOMEM;
+
+ ctx->dev = dev;
+ ctx->data = vop_data;
+ ctx->drm_dev = drm_dev;
+ dev_set_drvdata(dev, ctx);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ ctx->len = resource_size(res);
+ ctx->regs = devm_ioremap_resource(dev, res);
+ if (IS_ERR(ctx->regs))
+ return PTR_ERR(ctx->regs);
+
+ ctx->regsbak = devm_kzalloc(dev, ctx->len, GFP_KERNEL);
+ if (!ctx->regsbak)
+ return -ENOMEM;
+
+ ret = rockchip_vop_initial(ctx);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "cannot initial vop dev - err %d\n", ret);
+ return ret;
+ }
+
+ ctx->irq = platform_get_irq(pdev, 0);
+ if (ctx->irq < 0) {
+ dev_err(dev, "cannot find irq for vop\n");
+ return ctx->irq;
+ }
+
+ spin_lock_init(&ctx->reg_lock);
+ spin_lock_init(&ctx->irq_lock);
+
+ init_waitqueue_head(&ctx->wait_vsync_queue);
+ atomic_set(&ctx->wait_vsync_event, 0);
+
+ ret = vop_iommu_init(ctx);
+ if (ret) {
+ DRM_ERROR("Failed to setup iommu, %d\n", ret);
+ return ret;
+ }
+
+ ctx->vsync_wq = create_singlethread_workqueue("vsync");
+ if (!ctx->vsync_wq) {
+ dev_err(dev, "failed to create workqueue\n");
+ return -EINVAL;
+ }
+ INIT_WORK(&ctx->vsync_work, rockchip_vsync_worker);
+
+ mutex_init(&ctx->vsync_mutex);
+ pm_runtime_enable(&pdev->dev);
+
+ ret = devm_request_irq(dev, ctx->irq, rockchip_vop_isr,
+ IRQF_SHARED, dev_name(dev), ctx);
+ if (ret) {
+ dev_err(dev, "cannot requeset irq%d - err %d\n", ctx->irq, ret);
+ return ret;
+ }
+
+ return vop_create_crtc(ctx);
+}
+
+static void vop_unbind(struct device *dev, struct device *master,
+ void *data)
+{
+ struct drm_device *drm_dev = data;
+ struct vop_context *ctx = dev_get_drvdata(dev);
+ struct drm_crtc *crtc = &ctx->crtc;
+
+ drm_crtc_cleanup(crtc);
+ pm_runtime_disable(dev);
+ rockchip_drm_remove_crtc(drm_dev, ctx->pipe);
+
+ vop_iommu_fini(ctx);
+}
+
+static const struct component_ops vop_component_ops = {
+ .bind = vop_bind,
+ .unbind = vop_unbind,
+};
+
+static int vop_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct vop_context *ctx;
+
+ if (!dev->of_node) {
+ dev_err(dev, "can't find vop devices\n");
+ return -ENODEV;
+ }
+
+ platform_set_drvdata(pdev, ctx);
+
+ return component_add(dev, &vop_component_ops);
+}
+
+static int vop_remove(struct platform_device *pdev)
+{
+ component_del(&pdev->dev, &vop_component_ops);
+
+ return 0;
+}
+
+struct platform_driver rockchip_vop_platform_driver = {
+ .probe = vop_probe,
+ .remove = vop_remove,
+ .driver = {
+ .name = "rockchip-vop",
+ .owner = THIS_MODULE,
+ .of_match_table = of_match_ptr(vop_driver_dt_match),
+ },
+};
+
+module_platform_driver(rockchip_vop_platform_driver);
+
+MODULE_AUTHOR("Mark Yao <mark.yao@rock-chips.com>");
+MODULE_DESCRIPTION("ROCKCHIP VOP Driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_vop.h b/drivers/gpu/drm/rockchip/rockchip_drm_vop.h
new file mode 100644
index 0000000..2343760
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_vop.h
@@ -0,0 +1,187 @@
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Author:Mark Yao <mark.yao@rock-chips.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _ROCKCHIP_DRM_VOP_H
+#define _ROCKCHIP_DRM_VOP_H
+
+/* register definition */
+#define REG_CFG_DONE 0x0000
+#define VERSION_INFO 0x0004
+#define SYS_CTRL 0x0008
+#define SYS_CTRL1 0x000c
+#define DSP_CTRL0 0x0010
+#define DSP_CTRL1 0x0014
+#define DSP_BG 0x0018
+#define MCU_CTRL 0x001c
+#define INTR_CTRL0 0x0020
+#define INTR_CTRL1 0x0024
+#define WIN0_CTRL0 0x0030
+#define WIN0_CTRL1 0x0034
+#define WIN0_COLOR_KEY 0x0038
+#define WIN0_VIR 0x003c
+#define WIN0_YRGB_MST 0x0040
+#define WIN0_CBR_MST 0x0044
+#define WIN0_ACT_INFO 0x0048
+#define WIN0_DSP_INFO 0x004c
+#define WIN0_DSP_ST 0x0050
+#define WIN0_SCL_FACTOR_YRGB 0x0054
+#define WIN0_SCL_FACTOR_CBR 0x0058
+#define WIN0_SCL_OFFSET 0x005c
+#define WIN0_SRC_ALPHA_CTRL 0x0060
+#define WIN0_DST_ALPHA_CTRL 0x0064
+#define WIN0_FADING_CTRL 0x0068
+/* win1 register */
+#define WIN1_CTRL0 0x0070
+#define WIN1_CTRL1 0x0074
+#define WIN1_COLOR_KEY 0x0078
+#define WIN1_VIR 0x007c
+#define WIN1_YRGB_MST 0x0080
+#define WIN1_CBR_MST 0x0084
+#define WIN1_ACT_INFO 0x0088
+#define WIN1_DSP_INFO 0x008c
+#define WIN1_DSP_ST 0x0090
+#define WIN1_SCL_FACTOR_YRGB 0x0094
+#define WIN1_SCL_FACTOR_CBR 0x0098
+#define WIN1_SCL_OFFSET 0x009c
+#define WIN1_SRC_ALPHA_CTRL 0x00a0
+#define WIN1_DST_ALPHA_CTRL 0x00a4
+#define WIN1_FADING_CTRL 0x00a8
+/* win2 register */
+#define WIN2_CTRL0 0x00b0
+#define WIN2_CTRL1 0x00b4
+#define WIN2_VIR0_1 0x00b8
+#define WIN2_VIR2_3 0x00bc
+#define WIN2_MST0 0x00c0
+#define WIN2_DSP_INFO0 0x00c4
+#define WIN2_DSP_ST0 0x00c8
+#define WIN2_COLOR_KEY 0x00cc
+#define WIN2_MST1 0x00d0
+#define WIN2_DSP_INFO1 0x00d4
+#define WIN2_DSP_ST1 0x00d8
+#define WIN2_SRC_ALPHA_CTRL 0x00dc
+#define WIN2_MST2 0x00e0
+#define WIN2_DSP_INFO2 0x00e4
+#define WIN2_DSP_ST2 0x00e8
+#define WIN2_DST_ALPHA_CTRL 0x00ec
+#define WIN2_MST3 0x00f0
+#define WIN2_DSP_INFO3 0x00f4
+#define WIN2_DSP_ST3 0x00f8
+#define WIN2_FADING_CTRL 0x00fc
+/* win3 register */
+#define WIN3_CTRL0 0x0100
+#define WIN3_CTRL1 0x0104
+#define WIN3_VIR0_1 0x0108
+#define WIN3_VIR2_3 0x010c
+#define WIN3_MST0 0x0110
+#define WIN3_DSP_INFO0 0x0114
+#define WIN3_DSP_ST0 0x0118
+#define WIN3_COLOR_KEY 0x011c
+#define WIN3_MST1 0x0120
+#define WIN3_DSP_INFO1 0x0124
+#define WIN3_DSP_ST1 0x0128
+#define WIN3_SRC_ALPHA_CTRL 0x012c
+#define WIN3_MST2 0x0130
+#define WIN3_DSP_INFO2 0x0134
+#define WIN3_DSP_ST2 0x0138
+#define WIN3_DST_ALPHA_CTRL 0x013c
+#define WIN3_MST3 0x0140
+#define WIN3_DSP_INFO3 0x0144
+#define WIN3_DSP_ST3 0x0148
+#define WIN3_FADING_CTRL 0x014c
+/* hwc register */
+#define HWC_CTRL0 0x0150
+#define HWC_CTRL1 0x0154
+#define HWC_MST 0x0158
+#define HWC_DSP_ST 0x015c
+#define HWC_SRC_ALPHA_CTRL 0x0160
+#define HWC_DST_ALPHA_CTRL 0x0164
+#define HWC_FADING_CTRL 0x0168
+/* post process register */
+#define POST_DSP_HACT_INFO 0x0170
+#define POST_DSP_VACT_INFO 0x0174
+#define POST_SCL_FACTOR_YRGB 0x0178
+#define POST_SCL_CTRL 0x0180
+#define POST_DSP_VACT_INFO_F1 0x0184
+#define DSP_HTOTAL_HS_END 0x0188
+#define DSP_HACT_ST_END 0x018c
+#define DSP_VTOTAL_VS_END 0x0190
+#define DSP_VACT_ST_END 0x0194
+#define DSP_VS_ST_END_F1 0x0198
+#define DSP_VACT_ST_END_F1 0x019c
+/* register definition end */
+
+/* interrupt define */
+#define DSP_HOLD_VALID_INTR (1 << 0)
+#define FS_INTR (1 << 1)
+#define LINE_FLAG_INTR (1 << 2)
+#define BUS_ERROR_INTR (1 << 3)
+
+#define DSP_HOLD_VALID_INTR_EN(x) ((x) << 4)
+#define FS_INTR_EN(x) ((x) << 5)
+#define LINE_FLAG_INTR_EN(x) ((x) << 6)
+#define BUS_ERROR_INTR_EN(x) ((x) << 7)
+#define DSP_HOLD_VALID_INTR_MASK (1 << 4)
+#define FS_INTR_EN_MASK (1 << 5)
+#define LINE_FLAG_INTR_MASK (1 << 6)
+#define BUS_ERROR_INTR_MASK (1 << 7)
+
+#define DSP_HOLD_VALID_INTR_CLR (1 << 8)
+#define FS_INTR_EN_CLR (1 << 9)
+#define LINE_FLAG_INTR_CLR (1 << 10)
+#define BUS_ERROR_INTR_CLR (1 << 11)
+#define DSP_LINE_NUM(x) (((x) & 0x1fff) << 12)
+#define DSP_LINE_NUM_MASK (0x1fff << 12)
+
+/* src alpha ctrl define */
+#define SRC_FADING_VALUE(x) (((x) & 0xff) << 24)
+#define SRC_GLOBAL_ALPHA(x) (((x) & 0xff) << 16)
+#define SRC_FACTOR_M0(x) (((x) & 0x7) << 6)
+#define SRC_ALPHA_CAL_M0(x) (((x) & 0x1) << 5)
+#define SRC_BLEND_M0(x) (((x) & 0x3) << 3)
+#define SRC_ALPHA_M0(x) (((x) & 0x1) << 2)
+#define SRC_COLOR_M0(x) (((x) & 0x1) << 1)
+#define SRC_ALPHA_EN(x) (((x) & 0x1) << 0)
+/* dst alpha ctrl define */
+#define DST_FACTOR_M0(x) (((x) & 0x7) << 6)
+
+enum alpha_mode {
+ ALPHA_STRAIGHT,
+ ALPHA_INVERSE,
+};
+
+enum global_blend_mode {
+ ALPHA_GLOBAL,
+ ALPHA_PER_PIX,
+ ALPHA_PER_PIX_GLOBAL,
+};
+
+enum alpha_cal_mode {
+ ALPHA_SATURATION,
+ ALPHA_NO_SATURATION,
+};
+
+enum color_mode {
+ ALPHA_SRC_PRE_MUL,
+ ALPHA_SRC_NO_PRE_MUL,
+};
+
+enum factor_mode {
+ ALPHA_ZERO,
+ ALPHA_ONE,
+ ALPHA_SRC,
+ ALPHA_SRC_INVERSE,
+ ALPHA_SRC_GLOBAL,
+};
+
+#endif /* _ROCKCHIP_DRM_VOP_H */
diff --git a/include/uapi/drm/rockchip_drm.h b/include/uapi/drm/rockchip_drm.h
new file mode 100644
index 0000000..3193360
--- /dev/null
+++ b/include/uapi/drm/rockchip_drm.h
@@ -0,0 +1,75 @@
+/*
+ *
+ * Copyright (c) Fuzhou Rockchip Electronics Co.Ltd
+ * Authors:
+ * Mark Yao <yzq@rock-chips.com>
+ *
+ * base on exynos_drm.h
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ */
+
+#ifndef _UAPI_ROCKCHIP_DRM_H
+#define _UAPI_ROCKCHIP_DRM_H
+
+#include <drm/drm.h>
+
+/**
+ * User-desired buffer creation information structure.
+ *
+ * @size: user-desired memory allocation size.
+ * @flags: user request for setting memory type or cache attributes.
+ * @handle: returned a handle to created gem object.
+ * - this handle will be set by gem module of kernel side.
+ */
+struct drm_rockchip_gem_create {
+ uint64_t size;
+ uint32_t flags;
+ uint32_t handle;
+};
+
+/**
+ * A structure for getting buffer offset.
+ *
+ * @handle: a pointer to gem object created.
+ * @pad: just padding to be 64-bit aligned.
+ * @offset: relatived offset value of the memory region allocated.
+ * - this value should be set by user.
+ */
+struct drm_rockchip_gem_map_off {
+ uint32_t handle;
+ uint32_t pad;
+ uint64_t offset;
+};
+
+/**
+ * A structure to gem information.
+ *
+ * @handle: a handle to gem object created.
+ * @flags: flag value including memory type and cache attribute and
+ * this value would be set by driver.
+ * @size: size to memory region allocated by gem and this size would
+ * be set by driver.
+ */
+struct drm_rockchip_gem_info {
+ uint32_t handle;
+ uint32_t flags;
+ uint64_t size;
+};
+
+#define DRM_ROCKCHIP_GEM_CREATE 0x00
+#define DRM_ROCKCHIP_GEM_GET 0x01
+#define DRM_ROCKCHIP_GEM_MAP_OFFSET 0x02
+
+#define DRM_IOCTL_ROCKCHIP_GEM_CREATE DRM_IOWR(DRM_COMMAND_BASE + \
+ DRM_ROCKCHIP_GEM_CREATE, struct drm_rockchip_gem_create)
+
+#define DRM_IOCTL_ROCKCHIP_GEM_MAP_OFFSET DRM_IOWR(DRM_COMMAND_BASE + \
+ DRM_ROCKCHIP_GEM_MAP_OFFSET, struct drm_rockchip_gem_map_off)
+
+#define DRM_IOCTL_ROCKCHIP_GEM_GET DRM_IOWR(DRM_COMMAND_BASE + \
+ DRM_ROCKCHIP_GEM_GET, struct drm_rockchip_gem_info)
+#endif /* _UAPI_ROCKCHIP_DRM_H */
--
1.7.9.5
^ permalink raw reply related
* [PATCH v4 2/5] dt-bindings: video: Add for rockchip display subsytem
From: Mark yao @ 2014-09-22 10:50 UTC (permalink / raw)
To: heiko, Boris BREZILLON, David Airlie, Rob Clark, Rob Herring,
Pawel Moll, Mark Rutland, Ian Campbell, Kumar Gala, Randy Dunlap,
Grant Likely, Greg Kroah-Hartman, John Stultz, Rom Lemarchand
Cc: devicetree, linux-doc, linux-kernel, dri-devel, linux-api,
linux-rockchip, dianders, marcheu, dbehr, olof, djkurtz, xjq, kfx,
cym, cf, zyw, xxm, huangtao, kever.yang, yxj, wxt, xw, Mark yao
In-Reply-To: <1411382820-1615-1-git-send-email-mark.yao@rock-chips.com>
This add a display subsystem comprise the all display interface nodes.
Signed-off-by: Mark Yao <mark.yao@rock-chips.com>
---
Changes in v2:
- add DRM master device node to list all display nodes that comprise
the graphics subsystem.
Changes in v3: None
Changes in v4: None
.../devicetree/bindings/video/rockchip-drm.txt | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
create mode 100644 Documentation/devicetree/bindings/video/rockchip-drm.txt
diff --git a/Documentation/devicetree/bindings/video/rockchip-drm.txt b/Documentation/devicetree/bindings/video/rockchip-drm.txt
new file mode 100644
index 0000000..7fff582
--- /dev/null
+++ b/Documentation/devicetree/bindings/video/rockchip-drm.txt
@@ -0,0 +1,19 @@
+Rockchip DRM master device
+================================
+
+The Rockchip DRM master device is a virtual device needed to list all
+vop devices or other display interface nodes that comprise the
+graphics subsystem.
+
+Required properties:
+- compatible: Should be "rockchip,display-subsystem"
+- ports: Should contain a list of phandles pointing to display interface port
+ of vop devices. vop definitions as defined in
+ Documentation/devicetree/bindings/video/rockchip-vop.txt
+
+example:
+
+display-subsystem {
+ compatible = "rockchip,display-subsystem";
+ ports = <&vopl_out>, <&vopb_out>;
+};
--
1.7.9.5
^ permalink raw reply related
* [PATCH v4 3/5] dt-bindings: video: Add documentation for rockchip vop
From: Mark yao @ 2014-09-22 10:57 UTC (permalink / raw)
To: heiko-4mtYJXux2i+zQB+pC5nmwQ, Boris BREZILLON, David Airlie,
Rob Clark, Rob Herring, Pawel Moll, Mark Rutland, Ian Campbell,
Kumar Gala, Randy Dunlap, Grant Likely, Greg Kroah-Hartman,
John Stultz, Rom Lemarchand
Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-doc-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
dianders-F7+t8E8rja9g9hUCZPvPmw, marcheu-F7+t8E8rja9g9hUCZPvPmw,
dbehr-F7+t8E8rja9g9hUCZPvPmw, olof-nZhT3qVonbNeoWH0uzbU5w,
djkurtz-F7+t8E8rja9g9hUCZPvPmw, xjq-TNX95d0MmH7DzftRWevZcw,
kfx-TNX95d0MmH7DzftRWevZcw, cym-TNX95d0MmH7DzftRWevZcw,
cf-TNX95d0MmH7DzftRWevZcw, zyw-TNX95d0MmH7DzftRWevZcw,
xxm-TNX95d0MmH7DzftRWevZcw, huangtao-TNX95d0MmH7DzftRWevZcw,
kever.yang-TNX95d0MmH7DzftRWevZcw, yxj-TNX95d0MmH7DzftRWevZcw,
wxt-TNX95d0MmH7DzftRWevZcw, xw-TNX95d0MmH7DzftRWevZcw, Mark yao
In-Reply-To: <1411382820-1615-1-git-send-email-mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
This adds binding documentation for Rockchip SoC VOP driver.
Signed-off-by: Mark Yao <mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
---
Changes in v2:
- rename "lcdc" to "vop"
- add vop reset
- add iommu node
- add port for display-subsystem
Changes in v3: None
Changes in v4: None
.../devicetree/bindings/video/rockchip-vop.txt | 58 ++++++++++++++++++++
1 file changed, 58 insertions(+)
create mode 100644 Documentation/devicetree/bindings/video/rockchip-vop.txt
diff --git a/Documentation/devicetree/bindings/video/rockchip-vop.txt b/Documentation/devicetree/bindings/video/rockchip-vop.txt
new file mode 100644
index 0000000..d15351f
--- /dev/null
+++ b/Documentation/devicetree/bindings/video/rockchip-vop.txt
@@ -0,0 +1,58 @@
+device-tree bindings for rockchip soc display controller (vop)
+
+VOP (Visual Output Processor) is the Display Controller for the Rockchip
+series of SoCs which transfers the image data from a video memory
+buffer to an external LCD interface.
+
+Required properties:
+- compatible: value should be one of the following
+ "rockchip,rk3288-vop";
+
+- interrupts: should contain a list of all VOP IP block interrupts in the
+ order: VSYNC, LCD_SYSTEM. The interrupt specifier
+ format depends on the interrupt controller used.
+
+- clocks: must include clock specifiers corresponding to entries in the
+ clock-names property.
+
+- clock-names: Must contain
+ aclk_vop: for ddr buffer transfer.
+ hclk_vop: for ahb bus to R/W the phy regs.
+ dclk_vop: pixel clock.
+
+- resets: Must contain an entry for each entry in reset-names.
+ See ../reset/reset.txt for details.
+- reset-names: Must include the following entries:
+ - axi
+ - ahb
+ - dclk
+
+- iommus: required a iommu node
+
+- port: A port node with endpoint definitions as defined in
+ Documentation/devicetree/bindings/media/video-interfaces.txt.
+
+Example:
+SoC specific DT entry:
+ vopb: vopb@ff930000 {
+ compatible = "rockchip,rk3288-vop";
+ reg = <0xff930000 0x19c>;
+ interrupts = <GIC_SPI 15 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&cru ACLK_VOP0>, <&cru DCLK_VOP0>, <&cru HCLK_VOP0>;
+ clock-names = "aclk_vop", "dclk_vop", "hclk_vop";
+ resets = <&cru SRST_LCDC1_AXI>, <&cru SRST_LCDC1_AHB>, <&cru SRST_LCDC1_DCLK>;
+ reset-names = "axi", "ahb", "dclk";
+ iommus = <&vopb_mmu>;
+ vopb_out: port {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ vopb_out_edp: endpoint@0 {
+ reg = <0>;
+ remote-endpoint=<&edp_in_vopb>;
+ };
+ vopb_out_hdmi: endpoint@1 {
+ reg = <1>;
+ remote-endpoint=<&hdmi_in_vopb>;
+ };
+ };
+ };
--
1.7.9.5
^ permalink raw reply related
* [PATCH v4 4/5] dt-bindings: video: Add documentation for rockchip edp
From: Mark yao @ 2014-09-22 10:58 UTC (permalink / raw)
To: heiko-4mtYJXux2i+zQB+pC5nmwQ, Boris BREZILLON, David Airlie,
Rob Clark, Rob Herring, Pawel Moll, Mark Rutland, Ian Campbell,
Kumar Gala, Randy Dunlap, Grant Likely, Greg Kroah-Hartman,
John Stultz, Rom Lemarchand
Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-doc-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
dianders-F7+t8E8rja9g9hUCZPvPmw, marcheu-F7+t8E8rja9g9hUCZPvPmw,
dbehr-F7+t8E8rja9g9hUCZPvPmw, olof-nZhT3qVonbNeoWH0uzbU5w,
djkurtz-F7+t8E8rja9g9hUCZPvPmw, xjq-TNX95d0MmH7DzftRWevZcw,
kfx-TNX95d0MmH7DzftRWevZcw, cym-TNX95d0MmH7DzftRWevZcw,
cf-TNX95d0MmH7DzftRWevZcw, zyw-TNX95d0MmH7DzftRWevZcw,
xxm-TNX95d0MmH7DzftRWevZcw, huangtao-TNX95d0MmH7DzftRWevZcw,
kever.yang-TNX95d0MmH7DzftRWevZcw, yxj-TNX95d0MmH7DzftRWevZcw,
wxt-TNX95d0MmH7DzftRWevZcw, xw-TNX95d0MmH7DzftRWevZcw, Mark yao,
Jeff Chen
In-Reply-To: <1411382820-1615-1-git-send-email-mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
Add binding documentation for Rockchip SoC EDP driver.
Signed-off-by: Jeff Chen <jeff.chen-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
Signed-off-by: Mark Yao <mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
---
Changes in v2:
- add edp reset
- add panel node
- add port for display-subsystem
Changes in v3: None
Changes in v4: None
.../devicetree/bindings/video/rockchip-edp.txt | 50 ++++++++++++++++++++
1 file changed, 50 insertions(+)
create mode 100644 Documentation/devicetree/bindings/video/rockchip-edp.txt
diff --git a/Documentation/devicetree/bindings/video/rockchip-edp.txt b/Documentation/devicetree/bindings/video/rockchip-edp.txt
new file mode 100644
index 0000000..515e806
--- /dev/null
+++ b/Documentation/devicetree/bindings/video/rockchip-edp.txt
@@ -0,0 +1,50 @@
+Rockchip RK3288 EDP interface
+================================
+
+Required properties:
+- compatible: "rockchip,rk3288-edp";
+
+- reg: physical base address of the controller and length
+- clocks: from common clock binding: handle to dp clock.
+ of memory mapped region.
+- clock-names: from common clock binding:
+ Required elements: "clk_edp"
+ "clk_edp_24m"
+ "pclk_edp"
+- resets: Must contain an entry for each entry in reset-names.
+ See ../reset/reset.txt for details.
+- reset-names: Must include the name "edp"
+
+- rockchip,grf: this soc should set GRF regs, so need get grf here.
+- rockchip,panel: required a simple panel node as described by
+ Documentation/devicetree/bindings/panel/simple-panel.txt
+
+- ports: contain a port node with endpoint definitions as defined in
+ Documentation/devicetree/bindings/media/video-interfaces.txt.
+
+Example:
+ edp: edp@ff970000 {
+ compatible = "rockchip,rk3288-edp";
+ reg = <0xff970000 0x4000>;
+ interrupts = <GIC_SPI 98 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&cru SCLK_EDP>, <&cru SCLK_EDP_24M>, <&cru PCLK_EDP_CTRL>;
+ clock-names = "clk_edp", "clk_edp_24m", "pclk_edp";
+ rockchip,grf = <&grf>;
+ resets = <&cru 111>;
+ reset-names = "edp";
+ rockchip,panel = <&panel>;
+ ports {
+ edp_in: port {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ edp_in_vopb: endpoint@0 {
+ reg = <0>;
+ remote-endpoint = <&vopb_out_edp>;
+ };
+ edp_in_vopl: endpoint@1 {
+ reg = <1>;
+ remote-endpoint = <&vopl_out_edp>;
+ };
+ };
+ };
+ };
--
1.7.9.5
^ permalink raw reply related
* [PATCH v4 5/5] drm/rockchip: Add support for Rockchip Soc EDP
From: Mark yao @ 2014-09-22 11:02 UTC (permalink / raw)
To: heiko-4mtYJXux2i+zQB+pC5nmwQ, Boris BREZILLON, David Airlie,
Rob Clark, Rob Herring, Pawel Moll, Mark Rutland, Ian Campbell,
Kumar Gala, Randy Dunlap, Grant Likely, Greg Kroah-Hartman,
John Stultz, Rom Lemarchand
Cc: devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-doc-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
dianders-F7+t8E8rja9g9hUCZPvPmw, marcheu-F7+t8E8rja9g9hUCZPvPmw,
dbehr-F7+t8E8rja9g9hUCZPvPmw, olof-nZhT3qVonbNeoWH0uzbU5w,
djkurtz-F7+t8E8rja9g9hUCZPvPmw, xjq-TNX95d0MmH7DzftRWevZcw,
kfx-TNX95d0MmH7DzftRWevZcw, cym-TNX95d0MmH7DzftRWevZcw,
cf-TNX95d0MmH7DzftRWevZcw, zyw-TNX95d0MmH7DzftRWevZcw,
xxm-TNX95d0MmH7DzftRWevZcw, huangtao-TNX95d0MmH7DzftRWevZcw,
kever.yang-TNX95d0MmH7DzftRWevZcw, yxj-TNX95d0MmH7DzftRWevZcw,
wxt-TNX95d0MmH7DzftRWevZcw, xw-TNX95d0MmH7DzftRWevZcw, Mark yao,
Jeff Chen
In-Reply-To: <1411382820-1615-1-git-send-email-mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
This adds support for Rockchip soc edp found on rk3288
Signed-off-by: Mark Yao <mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
Signed-off-by: Jeff Chen <jeff.chen-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
---
Changes in v2:
- fix code sytle
- use some define from drm_dp_helper.h
- use panel-simple driver for primary display.
- remove unnecessary clock clk_24m_parent.
Changes in v3: None
Changes in v4: None
drivers/gpu/drm/rockchip/Kconfig | 9 +
drivers/gpu/drm/rockchip/Makefile | 2 +
drivers/gpu/drm/rockchip/rockchip_edp_core.c | 853 ++++++++++++++++++
drivers/gpu/drm/rockchip/rockchip_edp_core.h | 309 +++++++
drivers/gpu/drm/rockchip/rockchip_edp_reg.c | 1202 ++++++++++++++++++++++++++
drivers/gpu/drm/rockchip/rockchip_edp_reg.h | 345 ++++++++
6 files changed, 2720 insertions(+)
create mode 100644 drivers/gpu/drm/rockchip/rockchip_edp_core.c
create mode 100644 drivers/gpu/drm/rockchip/rockchip_edp_core.h
create mode 100644 drivers/gpu/drm/rockchip/rockchip_edp_reg.c
create mode 100644 drivers/gpu/drm/rockchip/rockchip_edp_reg.h
diff --git a/drivers/gpu/drm/rockchip/Kconfig b/drivers/gpu/drm/rockchip/Kconfig
index 7146c80..04b1f8c 100644
--- a/drivers/gpu/drm/rockchip/Kconfig
+++ b/drivers/gpu/drm/rockchip/Kconfig
@@ -17,3 +17,12 @@ config DRM_ROCKCHIP
management to userspace. This driver does not provides
2D or 3D acceleration; acceleration is performed by other
IP found on the SoC.
+
+config ROCKCHIP_EDP
+ bool "Rockchip edp support"
+ depends on DRM_ROCKCHIP
+ help
+ Choose this option if you have a Rockchip eDP.
+ Rockchip rk3288 SoC has eDP TX Controller can be used.
+ If you have an Embedded DisplayPort Panel, say Y to enable its
+ driver.
diff --git a/drivers/gpu/drm/rockchip/Makefile b/drivers/gpu/drm/rockchip/Makefile
index 6e6d468..a0fc3a1 100644
--- a/drivers/gpu/drm/rockchip/Makefile
+++ b/drivers/gpu/drm/rockchip/Makefile
@@ -7,4 +7,6 @@ ccflags-y := -Iinclude/drm -Idrivers/gpu/drm/rockchip
rockchipdrm-y := rockchip_drm_drv.o rockchip_drm_fb.o rockchip_drm_fbdev.o \
rockchip_drm_gem.o rockchip_drm_vop.o
+rockchipdrm-$(CONFIG_ROCKCHIP_EDP) += rockchip_edp_core.o rockchip_edp_reg.o
+
obj-$(CONFIG_DRM_ROCKCHIP) += rockchipdrm.o
diff --git a/drivers/gpu/drm/rockchip/rockchip_edp_core.c b/drivers/gpu/drm/rockchip/rockchip_edp_core.c
new file mode 100644
index 0000000..5450d1fa
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_edp_core.c
@@ -0,0 +1,853 @@
+/*
+* Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+* Author:
+* Andy yan <andy.yan-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
+* Jeff chen <jeff.chen-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
+*
+* based on exynos_dp_core.c
+*
+* This program is free software; you can redistribute it and/or modify it
+* under the terms of the GNU General Public License as published by the
+* Free Software Foundation; either version 2 of the License, or (at your
+* option) any later version.
+*/
+
+#include <drm/drmP.h>
+#include <drm/drm_crtc_helper.h>
+#include <drm/drm_panel.h>
+#include <drm/drm_of.h>
+
+#include <linux/component.h>
+#include <linux/clk.h>
+#include <linux/mfd/syscon.h>
+#include <linux/regmap.h>
+#include <linux/reset.h>
+
+#include <video/of_videomode.h>
+#include <video/videomode.h>
+
+#include "rockchip_edp_core.h"
+
+#define connector_to_edp(c) \
+ container_of(c, struct rockchip_edp_device, connector)
+
+#define encoder_to_edp(c) \
+ container_of(c, struct rockchip_edp_device, encoder)
+
+static struct rockchip_edp_soc_data soc_data[2] = {
+ /* rk3288 */
+ {.grf_soc_con6 = 0x025c,
+ .grf_soc_con12 = 0x0274},
+ /* no edp switching needed */
+ {.grf_soc_con6 = -1,
+ .grf_soc_con12 = -1},
+};
+
+static const struct of_device_id rockchip_edp_dt_ids[] = {
+ {.compatible = "rockchip,rk3288-edp",
+ .data = (void *)&soc_data[0] },
+ {}
+};
+
+MODULE_DEVICE_TABLE(of, rockchip_edp_dt_ids);
+
+static int rockchip_edp_clk_enable(struct rockchip_edp_device *edp)
+{
+ int ret = 0;
+
+ if (!edp->clk_on) {
+ ret = clk_prepare_enable(edp->pclk);
+ if (ret < 0) {
+ dev_err(edp->dev, "cannot enable edp pclk %d\n", ret);
+ goto err_pclk;
+ }
+
+ ret = clk_prepare_enable(edp->clk_edp);
+ if (ret < 0) {
+ dev_err(edp->dev, "cannot enable clk_edp %d\n", ret);
+ goto err_clk_edp;
+ }
+
+ ret = clk_set_rate(edp->clk_24m, 24000000);
+ if (ret < 0) {
+ dev_err(edp->dev, "cannot set edp clk_24m %d\n",
+ ret);
+ goto err_clk_24m;
+ }
+
+ ret = clk_prepare_enable(edp->clk_24m);
+ if (ret < 0) {
+ dev_err(edp->dev, "cannot enable edp clk_24m %d\n",
+ ret);
+ goto err_clk_24m;
+ }
+
+ edp->clk_on = true;
+ }
+
+ return 0;
+
+err_clk_24m:
+ clk_disable_unprepare(edp->clk_edp);
+err_clk_edp:
+ clk_disable_unprepare(edp->pclk);
+err_pclk:
+ edp->clk_on = false;
+
+ return ret;
+}
+
+static int rockchip_edp_clk_disable(struct rockchip_edp_device *edp)
+{
+ if (edp->clk_on) {
+ clk_disable_unprepare(edp->pclk);
+ clk_disable_unprepare(edp->clk_edp);
+ clk_disable_unprepare(edp->clk_24m);
+ edp->clk_on = false;
+ }
+
+ return 0;
+}
+
+static int rockchip_edp_pre_init(struct rockchip_edp_device *edp)
+{
+ u32 val;
+ int ret;
+
+ val = GRF_EDP_REF_CLK_SEL_INTER | (GRF_EDP_REF_CLK_SEL_INTER << 16);
+ ret = regmap_write(edp->grf, edp->soc_data->grf_soc_con12, val);
+ if (ret != 0) {
+ dev_err(edp->dev, "Could not write to GRF: %d\n", ret);
+ return ret;
+ }
+
+ reset_control_assert(edp->rst);
+ usleep_range(10, 20);
+ reset_control_deassert(edp->rst);
+
+ return 0;
+}
+
+static int rockchip_edp_init_edp(struct rockchip_edp_device *edp)
+{
+ rockchip_edp_reset(edp);
+ rockchip_edp_init_refclk(edp);
+ rockchip_edp_init_interrupt(edp);
+ rockchip_edp_enable_sw_function(edp);
+ rockchip_edp_init_analog_func(edp);
+ rockchip_edp_init_hpd(edp);
+ rockchip_edp_init_aux(edp);
+
+ return 0;
+}
+
+static int rockchip_edp_get_max_rx_bandwidth(
+ struct rockchip_edp_device *edp,
+ u8 *bandwidth)
+{
+ u8 data;
+ int retval;
+
+ /*
+ * For DP rev.1.1, Maximum link rate of Main Link lanes
+ * 0x06 = 1.62 Gbps, 0x0a = 2.7 Gbps
+ */
+ retval = rockchip_edp_read_byte_from_dpcd(
+ edp, DP_MAX_LINK_RATE, &data);
+ if (retval < 0)
+ *bandwidth = 0;
+ else
+ *bandwidth = data;
+
+ return retval;
+}
+
+static int rockchip_edp_get_max_rx_lane_count(struct rockchip_edp_device *edp,
+ u8 *lane_count)
+{
+ u8 data;
+ int retval;
+
+ /*
+ * For DP rev.1.1, Maximum number of Main Link lanes
+ * 0x01 = 1 lane, 0x02 = 2 lanes, 0x04 = 4 lanes
+ */
+ retval = rockchip_edp_read_byte_from_dpcd(
+ edp, DP_MAX_LANE_COUNT, &data);
+ if (retval < 0)
+ *lane_count = 0;
+ else
+ *lane_count = DPCD_MAX_LANE_COUNT(data);
+
+ return retval;
+}
+
+static int rockchip_edp_init_training(struct rockchip_edp_device *edp)
+{
+ int retval;
+
+ /*
+ * MACRO_RST must be applied after the PLL_LOCK to avoid
+ * the DP inter pair skew issue for at least 10 us
+ */
+ rockchip_edp_reset_macro(edp);
+
+ retval = rockchip_edp_get_max_rx_bandwidth(
+ edp, &edp->link_train.link_rate);
+ retval = rockchip_edp_get_max_rx_lane_count(
+ edp, &edp->link_train.lane_count);
+ dev_dbg(edp->dev, "max link rate:%d.%dGps max number of lanes:%d\n",
+ edp->link_train.link_rate * 27 / 100,
+ edp->link_train.link_rate * 27 % 100,
+ edp->link_train.lane_count);
+
+ if ((edp->link_train.link_rate != DP_LINK_BW_1_62) &&
+ (edp->link_train.link_rate != DP_LINK_BW_2_7)) {
+ dev_warn(edp->dev, "Rx Max Link Rate is abnormal :%x !\n"
+ "use default link rate:%d.%dGps\n",
+ edp->link_train.link_rate,
+ edp->video_info.link_rate * 27 / 100,
+ edp->video_info.link_rate * 27 % 100);
+ edp->link_train.link_rate = edp->video_info.link_rate;
+ }
+
+ if (edp->link_train.lane_count == 0) {
+ dev_err(edp->dev, "Rx Max Lane count is abnormal :%x !\n"
+ "use default lanes:%d\n",
+ edp->link_train.lane_count,
+ edp->video_info.lane_count);
+ edp->link_train.lane_count = edp->video_info.lane_count;
+ }
+
+ rockchip_edp_analog_power_ctr(edp, 1);
+
+ return 0;
+}
+
+static int rockchip_edp_hw_link_training(struct rockchip_edp_device *edp)
+{
+ u32 cnt = 50;
+ u32 val;
+
+ /* Set link rate and count as you want to establish*/
+ rockchip_edp_set_link_bandwidth(edp, edp->link_train.link_rate);
+ rockchip_edp_set_lane_count(edp, edp->link_train.lane_count);
+ rockchip_edp_hw_link_training_en(edp);
+ val = rockchip_edp_wait_hw_lt_done(edp);
+ while (val) {
+ if (cnt-- <= 0) {
+ dev_err(edp->dev, "hw lt timeout");
+ return -ETIMEDOUT;
+ }
+ mdelay(1);
+ val = rockchip_edp_wait_hw_lt_done(edp);
+ }
+
+ val = rockchip_edp_get_hw_lt_status(edp);
+ if (val)
+ dev_err(edp->dev, "hw lt err:%d\n", val);
+
+ return val;
+}
+
+static int rockchip_edp_set_link_train(struct rockchip_edp_device *edp)
+{
+ int retval;
+
+ rockchip_edp_init_training(edp);
+
+ retval = rockchip_edp_hw_link_training(edp);
+ if (retval < 0)
+ dev_err(edp->dev, "DP hw LT failed!\n");
+
+ return retval;
+}
+
+static int rockchip_edp_config_video(struct rockchip_edp_device *edp,
+ struct video_info *video_info)
+{
+ int retval = 0;
+ int timeout_loop = 0;
+ int done_count = 0;
+
+ rockchip_edp_config_video_slave_mode(edp, video_info);
+
+ rockchip_edp_set_video_color_format(edp, video_info->color_depth,
+ video_info->color_space,
+ video_info->dynamic_range,
+ video_info->ycbcr_coeff);
+
+ if (rockchip_edp_get_pll_lock_status(edp) == DP_PLL_UNLOCKED) {
+ dev_err(edp->dev, "PLL is not locked yet.\n");
+ return -EINVAL;
+ }
+
+ for (;;) {
+ timeout_loop++;
+ if (rockchip_edp_is_slave_video_stream_clock_on(edp) == 0)
+ break;
+
+ if (DP_TIMEOUT_LOOP_CNT < timeout_loop) {
+ dev_err(edp->dev, "Timeout of video streamclk ok\n");
+ return -ETIMEDOUT;
+ }
+
+ udelay(1);
+ }
+
+ /* Set to use the register calculated M/N video */
+ rockchip_edp_set_video_cr_mn(edp, CALCULATED_M, 0, 0);
+
+ /* Disable video mute */
+ rockchip_edp_enable_video_mute(edp, 0);
+
+ /* Configure video slave mode */
+ rockchip_edp_enable_video_master(edp, 0);
+
+ /* Enable video */
+ rockchip_edp_start_video(edp);
+
+ timeout_loop = 0;
+
+ for (;;) {
+ timeout_loop++;
+ if (rockchip_edp_is_video_stream_on(edp) == 0) {
+ done_count++;
+ if (done_count > 10)
+ break;
+ } else if (done_count) {
+ done_count = 0;
+ }
+ if (DP_TIMEOUT_LOOP_CNT < timeout_loop) {
+ dev_err(edp->dev, "Timeout of video streamclk ok\n");
+ return -ETIMEDOUT;
+ }
+
+ mdelay(1);
+ }
+
+ if (retval != 0)
+ dev_err(edp->dev, "Video stream is not detected!\n");
+
+ return retval;
+}
+
+static irqreturn_t rockchip_edp_isr(int irq, void *arg)
+{
+ struct rockchip_edp_device *edp = arg;
+ enum dp_irq_type irq_type;
+
+ irq_type = rockchip_edp_get_irq_type(edp);
+ switch (irq_type) {
+ case DP_IRQ_TYPE_HP_CABLE_IN:
+ dev_dbg(edp->dev, "Received irq - cable in\n");
+ rockchip_edp_clear_hotplug_interrupts(edp);
+ break;
+ case DP_IRQ_TYPE_HP_CABLE_OUT:
+ dev_dbg(edp->dev, "Received irq - cable out\n");
+ rockchip_edp_clear_hotplug_interrupts(edp);
+ break;
+ case DP_IRQ_TYPE_HP_CHANGE:
+ /*
+ * We get these change notifications once in a while, but there
+ * is nothing we can do with them. Just ignore it for now and
+ * only handle cable changes.
+ */
+ dev_dbg(edp->dev, "Received irq - hotplug change; ignoring.\n");
+ rockchip_edp_clear_hotplug_interrupts(edp);
+ break;
+ default:
+ dev_err(edp->dev, "Received irq - unknown type[%x]!\n",
+ irq_type);
+ rockchip_edp_clear_hotplug_interrupts(edp);
+ break;
+ }
+
+ return IRQ_HANDLED;
+}
+
+static void rockchip_edp_commit(struct drm_encoder *encoder)
+{
+ struct rockchip_edp_device *edp = encoder_to_edp(encoder);
+ int ret;
+
+ ret = rockchip_edp_set_link_train(edp);
+ if (ret)
+ dev_err(edp->dev, "link train failed!\n");
+ else
+ dev_dbg(edp->dev, "link training success.\n");
+
+ rockchip_edp_set_lane_count(edp, edp->link_train.lane_count);
+ rockchip_edp_set_link_bandwidth(edp, edp->link_train.link_rate);
+ rockchip_edp_init_video(edp);
+
+ ret = rockchip_edp_config_video(edp, &edp->video_info);
+ if (ret)
+ dev_err(edp->dev, "unable to config video\n");
+}
+
+static void rockchip_edp_poweron(struct drm_encoder *encoder)
+{
+ struct rockchip_edp_device *edp = encoder_to_edp(encoder);
+ int ret;
+
+ if (edp->dpms_mode == DRM_MODE_DPMS_ON)
+ return;
+
+ if (edp->panel)
+ edp->panel->funcs->enable(edp->panel);
+
+ ret = rockchip_edp_clk_enable(edp);
+ if (ret < 0) {
+ dev_err(edp->dev, "cannot enable edp clk %d\n", ret);
+ return;
+ }
+
+ ret = rockchip_edp_pre_init(edp);
+ if (ret < 0) {
+ dev_err(edp->dev, "edp pre init fail %d\n", ret);
+ return;
+ }
+
+ ret = rockchip_edp_init_edp(edp);
+ if (ret < 0) {
+ dev_err(edp->dev, "edp init fail %d\n", ret);
+ return;
+ }
+
+ enable_irq(edp->irq);
+ rockchip_edp_commit(encoder);
+}
+
+static void rockchip_edp_poweroff(struct drm_encoder *encoder)
+{
+ struct rockchip_edp_device *edp = encoder_to_edp(encoder);
+
+ if (edp->dpms_mode == DRM_MODE_DPMS_OFF)
+ return;
+
+ disable_irq(edp->irq);
+ rockchip_edp_reset(edp);
+ rockchip_edp_analog_power_ctr(edp, 0);
+ rockchip_edp_clk_disable(edp);
+ if (edp->panel)
+ edp->panel->funcs->disable(edp->panel);
+}
+
+static enum drm_connector_status
+rockchip_connector_detect(struct drm_connector *connector, bool force)
+{
+ return connector_status_connected;
+}
+
+static void rockchip_connector_destroy(struct drm_connector *connector)
+{
+ drm_sysfs_connector_remove(connector);
+ drm_connector_cleanup(connector);
+}
+
+static struct drm_connector_funcs rockchip_connector_funcs = {
+ .dpms = drm_helper_connector_dpms,
+ .detect = rockchip_connector_detect,
+ .fill_modes = drm_helper_probe_single_connector_modes,
+ .destroy = rockchip_connector_destroy,
+};
+
+static int rockchip_connector_get_modes(struct drm_connector *connector)
+{
+ struct rockchip_edp_device *edp = connector_to_edp(connector);
+ struct drm_panel *panel = edp->panel;
+
+ return panel->funcs->get_modes(panel);
+}
+
+static struct drm_encoder *
+ rockchip_connector_best_encoder(struct drm_connector *connector)
+{
+ struct rockchip_edp_device *edp = connector_to_edp(connector);
+
+ return &edp->encoder;
+}
+
+static enum drm_mode_status rockchip_connector_mode_valid(
+ struct drm_connector *connector,
+ struct drm_display_mode *mode)
+{
+ /* TODO(rk): verify that the mode is really valid */
+ return MODE_OK;
+}
+
+static struct drm_connector_helper_funcs rockchip_connector_helper_funcs = {
+ .get_modes = rockchip_connector_get_modes,
+ .mode_valid = rockchip_connector_mode_valid,
+ .best_encoder = rockchip_connector_best_encoder,
+};
+
+static void rockchip_drm_encoder_dpms(struct drm_encoder *encoder, int mode)
+{
+ struct rockchip_edp_device *edp = encoder_to_edp(encoder);
+
+ if (edp->dpms_mode == mode)
+ return;
+
+ switch (mode) {
+ case DRM_MODE_DPMS_ON:
+ rockchip_edp_poweron(encoder);
+ break;
+ case DRM_MODE_DPMS_STANDBY:
+ case DRM_MODE_DPMS_SUSPEND:
+ case DRM_MODE_DPMS_OFF:
+ rockchip_edp_poweroff(encoder);
+ break;
+ default:
+ break;
+ }
+
+ edp->dpms_mode = mode;
+}
+
+static bool
+rockchip_drm_encoder_mode_fixup(struct drm_encoder *encoder,
+ const struct drm_display_mode *mode,
+ struct drm_display_mode *adjusted_mode)
+{
+ if (!adjusted_mode->private) {
+ struct rockchip_display_mode *priv_mode;
+
+ priv_mode = kzalloc(sizeof(*priv_mode), GFP_KERNEL);
+ priv_mode->out_type = ROCKCHIP_DISPLAY_TYPE_EDP;
+ adjusted_mode->private = (int *)priv_mode;
+ }
+
+ return true;
+}
+
+static void rockchip_drm_encoder_mode_set(struct drm_encoder *encoder,
+ struct drm_display_mode *mode,
+ struct drm_display_mode *adjusted)
+{
+ struct rockchip_edp_device *edp = encoder_to_edp(encoder);
+ u32 val;
+ int ret;
+
+ ret = rockchip_drm_encoder_get_mux_id(edp->dev->of_node, encoder);
+ if (ret < 0)
+ return;
+
+ if (ret == ROCKCHIP_CRTC_VOPL)
+ val = EDP_SEL_VOP_LIT | (EDP_SEL_VOP_LIT << 16);
+ else
+ val = EDP_SEL_VOP_LIT << 16;
+
+ dev_info(edp->dev, "vop %s output to edp\n",
+ (ret == ROCKCHIP_CRTC_VOPL) ? "LIT" : "BIG");
+ ret = regmap_write(edp->grf, edp->soc_data->grf_soc_con6, val);
+ if (ret != 0) {
+ dev_err(edp->dev, "Could not write to GRF: %d\n", ret);
+ return;
+ }
+
+ memcpy(&edp->mode, adjusted, sizeof(*mode));
+}
+
+static void rockchip_drm_encoder_prepare(struct drm_encoder *encoder)
+{
+}
+
+static void rockchip_drm_encoder_commit(struct drm_encoder *encoder)
+{
+ rockchip_drm_encoder_dpms(encoder, DRM_MODE_DPMS_ON);
+}
+
+static void rockchip_drm_encoder_disable(struct drm_encoder *encoder)
+{
+ struct drm_plane *plane;
+ struct drm_device *dev = encoder->dev;
+
+ rockchip_drm_encoder_dpms(encoder, DRM_MODE_DPMS_OFF);
+
+ /* all planes connected to this encoder should be also disabled. */
+ list_for_each_entry(plane, &dev->mode_config.plane_list, head) {
+ if (plane->crtc && (plane->crtc == encoder->crtc))
+ plane->funcs->disable_plane(plane);
+ }
+}
+
+static struct drm_encoder_helper_funcs rockchip_encoder_helper_funcs = {
+ .dpms = rockchip_drm_encoder_dpms,
+ .mode_fixup = rockchip_drm_encoder_mode_fixup,
+ .mode_set = rockchip_drm_encoder_mode_set,
+ .prepare = rockchip_drm_encoder_prepare,
+ .commit = rockchip_drm_encoder_commit,
+ .disable = rockchip_drm_encoder_disable,
+};
+
+static void rockchip_drm_encoder_destroy(struct drm_encoder *encoder)
+{
+ drm_encoder_cleanup(encoder);
+}
+
+static struct drm_encoder_funcs rockchip_encoder_funcs = {
+ .destroy = rockchip_drm_encoder_destroy,
+};
+
+static int rockchip_edp_init(struct rockchip_edp_device *edp)
+{
+ struct device *dev = edp->dev;
+ struct device_node *np = dev->of_node;
+ struct platform_device *pdev = to_platform_device(dev);
+ struct resource *res;
+ const struct of_device_id *match;
+ int ret;
+
+ if (!np) {
+ dev_err(dev, "Missing device tree node.\n");
+ return -EINVAL;
+ }
+
+ match = of_match_node(rockchip_edp_dt_ids, np);
+ edp->soc_data = (struct rockchip_edp_soc_data *)match->data;
+ /*
+ * The control bit is located in the GRF register space.
+ */
+ if (edp->soc_data->grf_soc_con6 >= 0) {
+ edp->grf = syscon_regmap_lookup_by_phandle(np, "rockchip,grf");
+ if (IS_ERR(edp->grf)) {
+ dev_err(dev,
+ "rk3288-edp needs rockchip,grf property\n");
+ return PTR_ERR(edp->grf);
+ }
+ }
+
+ edp->video_info.h_sync_polarity = 0;
+ edp->video_info.v_sync_polarity = 0;
+ edp->video_info.interlaced = 0;
+ edp->video_info.color_space = CS_RGB;
+ edp->video_info.dynamic_range = VESA;
+ edp->video_info.ycbcr_coeff = COLOR_YCBCR601;
+ edp->video_info.color_depth = COLOR_8;
+
+ edp->video_info.link_rate = DP_LINK_BW_1_62;
+ edp->video_info.lane_count = LANE_CNT4;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ edp->regs = devm_ioremap_resource(dev, res);
+ if (IS_ERR(edp->regs)) {
+ dev_err(dev, "ioremap reg failed\n");
+ return PTR_ERR(edp->regs);
+ }
+
+ edp->clk_edp = devm_clk_get(dev, "clk_edp");
+ if (IS_ERR(edp->clk_edp)) {
+ dev_err(dev, "cannot get clk_edp\n");
+ return PTR_ERR(edp->clk_edp);
+ }
+
+ edp->clk_24m = devm_clk_get(dev, "clk_edp_24m");
+ if (IS_ERR(edp->clk_24m)) {
+ dev_err(dev, "cannot get clk_edp_24m\n");
+ return PTR_ERR(edp->clk_24m);
+ }
+
+ edp->pclk = devm_clk_get(dev, "pclk_edp");
+ if (IS_ERR(edp->pclk)) {
+ dev_err(dev, "cannot get pclk\n");
+ return PTR_ERR(edp->pclk);
+ }
+
+ edp->rst = devm_reset_control_get(dev, "edp");
+ if (IS_ERR(edp->rst)) {
+ dev_err(dev, "failed to get reset\n");
+ return PTR_ERR(edp->rst);
+ }
+
+ ret = rockchip_edp_clk_enable(edp);
+ if (ret < 0) {
+ dev_err(edp->dev, "cannot enable edp clk %d\n", ret);
+ return ret;
+ }
+
+ ret = rockchip_edp_pre_init(edp);
+ if (ret < 0) {
+ dev_err(edp->dev, "failed to pre init %d\n", ret);
+ return ret;
+ }
+
+ edp->irq = platform_get_irq(pdev, 0);
+ if (edp->irq < 0) {
+ dev_err(dev, "cannot find IRQ\n");
+ return edp->irq;
+ }
+
+ ret = devm_request_irq(dev, edp->irq, rockchip_edp_isr, 0,
+ dev_name(dev), edp);
+ if (ret) {
+ dev_err(dev, "cannot claim IRQ %d\n", edp->irq);
+ return ret;
+ }
+
+ disable_irq_nosync(edp->irq);
+
+ edp->dpms_mode = DRM_MODE_DPMS_OFF;
+
+ dev_set_name(edp->dev, "rockchip-edp");
+
+ return 0;
+}
+
+static int rockchip_edp_bind(struct device *dev, struct device *master,
+ void *data)
+{
+ struct rockchip_edp_device *edp = dev_get_drvdata(dev);
+ struct drm_encoder *encoder;
+ struct drm_connector *connector;
+ struct drm_device *drm_dev = data;
+ int ret;
+
+ ret = rockchip_edp_init(edp);
+ if (ret < 0)
+ return ret;
+
+ edp->drm_dev = drm_dev;
+
+ encoder = &edp->encoder;
+
+ encoder->possible_crtcs = drm_of_find_possible_crtcs(drm_dev,
+ dev->of_node);
+ DRM_DEBUG_KMS("possible_crtcs = 0x%x\n", encoder->possible_crtcs);
+
+ ret = drm_encoder_init(drm_dev, encoder, &rockchip_encoder_funcs,
+ DRM_MODE_ENCODER_LVDS);
+ if (ret) {
+ DRM_ERROR("failed to initialize encoder with drm\n");
+ return ret;
+ }
+
+ drm_encoder_helper_add(encoder, &rockchip_encoder_helper_funcs);
+
+ connector = &edp->connector;
+ connector->polled = DRM_CONNECTOR_POLL_HPD;
+ connector->dpms = DRM_MODE_DPMS_OFF;
+
+ ret = drm_connector_init(drm_dev, connector,
+ &rockchip_connector_funcs,
+ DRM_MODE_CONNECTOR_eDP);
+ if (ret) {
+ DRM_ERROR("failed to initialize connector with drm\n");
+ goto err_free_encoder;
+ }
+
+ drm_connector_helper_add(connector,
+ &rockchip_connector_helper_funcs);
+
+ ret = drm_sysfs_connector_add(connector);
+ if (ret) {
+ DRM_ERROR("failed to add drm_sysfs\n");
+ goto err_free_connector;
+ }
+
+ ret = drm_mode_connector_attach_encoder(connector, encoder);
+ if (ret) {
+ DRM_ERROR("failed to attach connector and encoder\n");
+ goto err_free_connector_sysfs;
+ }
+
+ ret = drm_panel_attach(edp->panel, connector);
+ if (ret) {
+ DRM_ERROR("failed to attach connector and encoder\n");
+ goto err_free_connector_sysfs;
+ }
+
+ return 0;
+
+err_free_connector_sysfs:
+ drm_sysfs_connector_remove(connector);
+err_free_connector:
+ drm_connector_cleanup(connector);
+err_free_encoder:
+ drm_encoder_cleanup(encoder);
+ return ret;
+}
+
+static void rockchip_edp_unbind(struct device *dev, struct device *master,
+ void *data)
+{
+ struct rockchip_edp_device *edp = dev_get_drvdata(dev);
+ struct drm_encoder *encoder;
+
+ encoder = &edp->encoder;
+
+ if (edp->panel)
+ drm_panel_detach(edp->panel);
+
+ rockchip_drm_encoder_dpms(encoder, DRM_MODE_DPMS_OFF);
+ encoder->funcs->destroy(encoder);
+ drm_sysfs_connector_remove(&edp->connector);
+ drm_connector_cleanup(&edp->connector);
+ drm_encoder_cleanup(encoder);
+}
+
+static const struct component_ops rockchip_edp_component_ops = {
+ .bind = rockchip_edp_bind,
+ .unbind = rockchip_edp_unbind,
+};
+
+static int rockchip_edp_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct drm_panel *panel;
+ struct device_node *panel_node;
+ struct rockchip_edp_device *edp;
+
+ if (!dev->of_node) {
+ dev_err(dev, "can't find eDP devices\n");
+ return -ENODEV;
+ }
+
+ panel_node = of_parse_phandle(dev->of_node, "rockchip,panel", 0);
+ if (!panel_node) {
+ DRM_ERROR("failed to find diaplay panel\n");
+ return -ENODEV;
+ }
+
+ panel = of_drm_find_panel(panel_node);
+ if (!panel) {
+ DRM_ERROR("failed to find diaplay panel\n");
+ of_node_put(panel_node);
+ return -EPROBE_DEFER;
+ }
+
+ of_node_put(panel_node);
+
+ edp = devm_kzalloc(dev, sizeof(*edp), GFP_KERNEL);
+ if (!edp)
+ return -ENOMEM;
+ edp->dev = dev;
+ edp->panel = panel;
+ platform_set_drvdata(pdev, edp);
+
+ return component_add(dev, &rockchip_edp_component_ops);
+}
+
+static int rockchip_edp_remove(struct platform_device *pdev)
+{
+ component_del(&pdev->dev, &rockchip_edp_component_ops);
+
+ return 0;
+}
+
+static struct platform_driver rockchip_edp_driver = {
+ .probe = rockchip_edp_probe,
+ .remove = rockchip_edp_remove,
+ .driver = {
+ .name = "rockchip-edp",
+ .owner = THIS_MODULE,
+ .of_match_table = of_match_ptr(rockchip_edp_dt_ids),
+ },
+};
+
+module_platform_driver(rockchip_edp_driver);
+
+MODULE_AUTHOR("Jeff chen <jeff.chen-TNX95d0MmH7DzftRWevZcw@public.gmane.org>");
+MODULE_DESCRIPTION("ROCKCHIP EDP Driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/gpu/drm/rockchip/rockchip_edp_core.h b/drivers/gpu/drm/rockchip/rockchip_edp_core.h
new file mode 100644
index 0000000..c13325f
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_edp_core.h
@@ -0,0 +1,309 @@
+/*
+* Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+* Author:
+* Andy yan <andy.yan-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
+* Jeff chen <jeff.chen-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
+*
+* based on exynos_dp_core.h
+*
+* This program is free software; you can redistribute it and/or modify it
+* under the terms of the GNU General Public License as published by the
+* Free Software Foundation; either version 2 of the License, or (at your
+* option) any later version.
+*/
+
+#ifndef _ROCKCHIP_EDP_CORE_H
+#define _ROCKCHIP_EDP_CORE_H
+
+#include <drm/drmP.h>
+#include <drm/drm_crtc_helper.h>
+#include <drm/drm_dp_helper.h>
+#include <drm/drm_panel.h>
+#include "rockchip_drm_drv.h"
+
+#define DP_TIMEOUT_LOOP_CNT 100
+#define MAX_CR_LOOP 5
+#define MAX_EQ_LOOP 5
+
+#define GRF_EDP_REF_CLK_SEL_INTER (1 << 4)
+#define GRF_EDP_HDCP_EN (1 << 15)
+#define GRF_EDP_BIST_EN (1 << 14)
+#define GRF_EDP_MEM_CTL_BY_EDP (1 << 13)
+#define GRF_EDP_SECURE_EN (1 << 3)
+#define EDP_SEL_VOP_LIT (1 << 5)
+
+enum link_lane_count_type {
+ LANE_CNT1 = 1,
+ LANE_CNT2 = 2,
+ LANE_CNT4 = 4
+};
+
+enum link_training_state {
+ LT_START,
+ LT_CLK_RECOVERY,
+ LT_EQ_TRAINING,
+ FINISHED,
+ FAILED
+};
+
+enum voltage_swing_level {
+ VOLTAGE_LEVEL_0,
+ VOLTAGE_LEVEL_1,
+ VOLTAGE_LEVEL_2,
+ VOLTAGE_LEVEL_3,
+};
+
+enum pre_emphasis_level {
+ PRE_EMPHASIS_LEVEL_0,
+ PRE_EMPHASIS_LEVEL_1,
+ PRE_EMPHASIS_LEVEL_2,
+ PRE_EMPHASIS_LEVEL_3,
+};
+
+enum pattern_set {
+ PRBS7,
+ D10_2,
+ TRAINING_PTN1,
+ TRAINING_PTN2,
+ DP_NONE
+};
+
+enum color_space {
+ CS_RGB,
+ CS_YCBCR422,
+ CS_YCBCR444
+};
+
+enum color_depth {
+ COLOR_6,
+ COLOR_8,
+ COLOR_10,
+ COLOR_12
+};
+
+enum color_coefficient {
+ COLOR_YCBCR601,
+ COLOR_YCBCR709
+};
+
+enum dynamic_range {
+ VESA,
+ CEA
+};
+
+enum pll_status {
+ DP_PLL_UNLOCKED,
+ DP_PLL_LOCKED
+};
+
+enum clock_recovery_m_value_type {
+ CALCULATED_M,
+ REGISTER_M
+};
+
+enum video_timing_recognition_type {
+ VIDEO_TIMING_FROM_CAPTURE,
+ VIDEO_TIMING_FROM_REGISTER
+};
+
+enum analog_power_block {
+ AUX_BLOCK,
+ CH0_BLOCK,
+ CH1_BLOCK,
+ CH2_BLOCK,
+ CH3_BLOCK,
+ ANALOG_TOTAL,
+ POWER_ALL
+};
+
+enum dp_irq_type {
+ DP_IRQ_TYPE_HP_CABLE_IN,
+ DP_IRQ_TYPE_HP_CABLE_OUT,
+ DP_IRQ_TYPE_HP_CHANGE,
+ DP_IRQ_TYPE_UNKNOWN,
+};
+
+struct video_info {
+ char *name;
+
+ bool h_sync_polarity;
+ bool v_sync_polarity;
+ bool interlaced;
+
+ enum color_space color_space;
+ enum dynamic_range dynamic_range;
+ enum color_coefficient ycbcr_coeff;
+ enum color_depth color_depth;
+
+ u8 link_rate;
+ enum link_lane_count_type lane_count;
+};
+
+struct link_train {
+ int eq_loop;
+ int cr_loop[4];
+
+ u8 link_rate;
+ u8 lane_count;
+ u8 training_lane[4];
+
+ enum link_training_state lt_state;
+};
+
+/*
+ * @grf_offset: offset inside the grf regmap for setting the rk3288 lvds
+ */
+struct rockchip_edp_soc_data {
+ int grf_soc_con6;
+ int grf_soc_con12;
+};
+
+struct rockchip_edp_device {
+ struct device *dev;
+ struct drm_device *drm_dev;
+ struct drm_panel *panel;
+ struct drm_connector connector;
+ struct drm_encoder encoder;
+ struct drm_display_mode mode;
+
+ struct rockchip_edp_soc_data *soc_data;
+
+ void __iomem *regs;
+ struct regmap *grf;
+ unsigned int irq;
+ struct clk *clk_edp;
+ struct clk *clk_24m_parent;
+ struct clk *clk_24m;
+ struct clk *pclk;
+ struct reset_control *rst;
+ struct link_train link_train;
+ struct video_info video_info;
+ bool clk_on;
+
+ int dpms_mode;
+};
+
+void rockchip_edp_enable_video_mute(struct rockchip_edp_device *edp,
+ bool enable);
+void rockchip_edp_stop_video(struct rockchip_edp_device *edp);
+void rockchip_edp_lane_swap(struct rockchip_edp_device *edp, bool enable);
+void rockchip_edp_init_refclk(struct rockchip_edp_device *edp);
+void rockchip_edp_init_interrupt(struct rockchip_edp_device *edp);
+void rockchip_edp_reset(struct rockchip_edp_device *edp);
+void rockchip_edp_config_interrupt(struct rockchip_edp_device *edp);
+u32 rockchip_edp_get_pll_lock_status(struct rockchip_edp_device *edp);
+void rockchip_edp_analog_power_ctr(struct rockchip_edp_device *edp,
+ bool enable);
+void rockchip_edp_init_analog_func(struct rockchip_edp_device *edp);
+void rockchip_edp_init_hpd(struct rockchip_edp_device *edp);
+void rockchip_edp_reset_aux(struct rockchip_edp_device *edp);
+void rockchip_edp_init_aux(struct rockchip_edp_device *edp);
+int rockchip_edp_get_plug_in_status(struct rockchip_edp_device *edp);
+void rockchip_edp_enable_sw_function(struct rockchip_edp_device *edp);
+int rockchip_edp_start_aux_transaction(struct rockchip_edp_device *edp);
+int rockchip_edp_write_byte_to_dpcd(struct rockchip_edp_device *edp,
+ unsigned int reg_addr,
+ unsigned char data);
+int rockchip_edp_read_byte_from_dpcd(struct rockchip_edp_device *edp,
+ unsigned int reg_addr,
+ unsigned char *data);
+int rockchip_edp_write_bytes_to_dpcd(struct rockchip_edp_device *edp,
+ unsigned int reg_addr,
+ unsigned int count,
+ unsigned char data[]);
+int rockchip_edp_read_bytes_from_dpcd(struct rockchip_edp_device *edp,
+ unsigned int reg_addr,
+ unsigned int count,
+ unsigned char data[]);
+int rockchip_edp_select_i2c_device(struct rockchip_edp_device *edp,
+ unsigned int device_addr,
+ unsigned int reg_addr);
+int rockchip_edp_read_byte_from_i2c(struct rockchip_edp_device *edp,
+ unsigned int device_addr,
+ unsigned int reg_addr,
+ unsigned int *data);
+int rockchip_edp_read_bytes_from_i2c(struct rockchip_edp_device *edp,
+ unsigned int device_addr,
+ unsigned int reg_addr,
+ unsigned int count,
+ unsigned char edid[]);
+void rockchip_edp_set_link_bandwidth(struct rockchip_edp_device *edp,
+ u32 bwtype);
+void rockchip_edp_get_link_bandwidth(struct rockchip_edp_device *edp,
+ u32 *bwtype);
+void rockchip_edp_set_lane_count(struct rockchip_edp_device *edp,
+ u32 count);
+void rockchip_edp_get_lane_count(struct rockchip_edp_device *edp,
+ u32 *count);
+void rockchip_edp_enable_enhanced_mode(struct rockchip_edp_device *edp,
+ bool enable);
+void rockchip_edp_set_training_pattern(struct rockchip_edp_device *edp,
+ enum pattern_set pattern);
+void rockchip_edp_set_lane0_pre_emphasis(struct rockchip_edp_device *edp,
+ u32 level);
+void rockchip_edp_set_lane1_pre_emphasis(struct rockchip_edp_device *edp,
+ u32 level);
+void rockchip_edp_set_lane2_pre_emphasis(struct rockchip_edp_device *edp,
+ u32 level);
+void rockchip_edp_set_lane3_pre_emphasis(struct rockchip_edp_device *edp,
+ u32 level);
+void rockchip_edp_set_lane0_link_training(struct rockchip_edp_device *edp,
+ u32 training_lane);
+void rockchip_edp_set_lane1_link_training(struct rockchip_edp_device *edp,
+ u32 training_lane);
+void rockchip_edp_set_lane2_link_training(struct rockchip_edp_device *edp,
+ u32 training_lane);
+void rockchip_edp_set_lane3_link_training(struct rockchip_edp_device *edp,
+ u32 training_lane);
+u32 rockchip_edp_get_lane0_link_training(struct rockchip_edp_device *edp);
+u32 rockchip_edp_get_lane1_link_training(struct rockchip_edp_device *edp);
+u32 rockchip_edp_get_lane2_link_training(struct rockchip_edp_device *edp);
+u32 rockchip_edp_get_lane3_link_training(struct rockchip_edp_device *edp);
+void rockchip_edp_reset_macro(struct rockchip_edp_device *edp);
+int rockchip_edp_init_video(struct rockchip_edp_device *edp);
+
+void rockchip_edp_set_video_color_format(struct rockchip_edp_device *edp,
+ u32 color_depth,
+ u32 color_space,
+ u32 dynamic_range,
+ u32 coeff);
+int
+rockchip_edp_is_slave_video_stream_clock_on(struct rockchip_edp_device *edp);
+void rockchip_edp_set_video_cr_mn(struct rockchip_edp_device *edp,
+ enum clock_recovery_m_value_type type,
+ u32 m_value,
+ u32 n_value);
+void rockchip_edp_set_video_timing_mode(struct rockchip_edp_device *edp,
+ u32 type);
+void rockchip_edp_enable_video_master(struct rockchip_edp_device *edp,
+ bool enable);
+void rockchip_edp_start_video(struct rockchip_edp_device *edp);
+int rockchip_edp_is_video_stream_on(struct rockchip_edp_device *edp);
+void rockchip_edp_config_video_slave_mode(struct rockchip_edp_device *edp,
+ struct video_info *video_info);
+void rockchip_edp_enable_scrambling(struct rockchip_edp_device *edp);
+void rockchip_edp_disable_scrambling(struct rockchip_edp_device *edp);
+void rockchip_edp_hw_link_training_en(struct rockchip_edp_device *edp);
+int rockchip_edp_get_hw_lt_status(struct rockchip_edp_device *edp);
+int rockchip_edp_wait_hw_lt_done(struct rockchip_edp_device *edp);
+enum dp_irq_type rockchip_edp_get_irq_type(struct rockchip_edp_device *edp);
+void rockchip_edp_clear_hotplug_interrupts(struct rockchip_edp_device *edp);
+
+/* I2C EDID Chip ID, Slave Address */
+#define I2C_EDID_DEVICE_ADDR 0x50
+#define I2C_E_EDID_DEVICE_ADDR 0x30
+
+/* DPCD_ADDR_MAX_LANE_COUNT */
+#define DPCD_ENHANCED_FRAME_CAP(x) (((x) >> 7) & 0x1)
+#define DPCD_MAX_LANE_COUNT(x) ((x) & 0x1f)
+
+/* DPCD_ADDR_LANE_COUNT_SET */
+#define DPCD_LANE_COUNT_SET(x) ((x) & 0x1f)
+
+/* DPCD_ADDR_TRAINING_LANE0_SET */
+#define DPCD_PRE_EMPHASIS_SET(x) (((x) & 0x3) << 3)
+#define DPCD_PRE_EMPHASIS_GET(x) (((x) >> 3) & 0x3)
+#define DPCD_VOLTAGE_SWING_SET(x) (((x) & 0x3) << 0)
+#define DPCD_VOLTAGE_SWING_GET(x) (((x) >> 0) & 0x3)
+
+#endif /* _ROCKCHIP_EDP_CORE_H */
diff --git a/drivers/gpu/drm/rockchip/rockchip_edp_reg.c b/drivers/gpu/drm/rockchip/rockchip_edp_reg.c
new file mode 100644
index 0000000..f6d641c
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_edp_reg.c
@@ -0,0 +1,1202 @@
+/*
+* Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+* Author:
+* Andy yan <andy.yan-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
+* Jeff chen <jeff.chen-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
+*
+* based on exynos_dp_reg.c
+*
+* This program is free software; you can redistribute it and/or modify it
+* under the terms of the GNU General Public License as published by the
+* Free Software Foundation; either version 2 of the License, or (at your
+* option) any later version.
+*/
+
+#include <linux/device.h>
+#include <linux/delay.h>
+#include <linux/io.h>
+
+#include "rockchip_edp_core.h"
+#include "rockchip_edp_reg.h"
+
+void rockchip_edp_enable_video_mute(struct rockchip_edp_device *edp,
+ bool enable)
+{
+ u32 val;
+
+ if (enable) {
+ val = readl(edp->regs + VIDEO_CTL_1);
+ val |= VIDEO_MUTE;
+ writel(val, edp->regs + VIDEO_CTL_1);
+ } else {
+ val = readl(edp->regs + VIDEO_CTL_1);
+ val &= ~VIDEO_MUTE;
+ writel(val, edp->regs + VIDEO_CTL_1);
+ }
+}
+
+void rockchip_edp_stop_video(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + VIDEO_CTL_1);
+ val &= ~VIDEO_EN;
+ writel(val, edp->regs + VIDEO_CTL_1);
+}
+
+void rockchip_edp_lane_swap(struct rockchip_edp_device *edp, bool enable)
+{
+ u32 val;
+
+ if (enable)
+ val = LANE3_MAP_LOGIC_LANE_0 | LANE2_MAP_LOGIC_LANE_1 |
+ LANE1_MAP_LOGIC_LANE_2 | LANE0_MAP_LOGIC_LANE_3;
+ else
+ val = LANE3_MAP_LOGIC_LANE_3 | LANE2_MAP_LOGIC_LANE_2 |
+ LANE1_MAP_LOGIC_LANE_1 | LANE0_MAP_LOGIC_LANE_0;
+
+ writel(val, edp->regs + LANE_MAP);
+}
+
+void rockchip_edp_init_refclk(struct rockchip_edp_device *edp)
+{
+ writel(SEL_24M, edp->regs + ANALOG_CTL_2);
+ writel(REF_CLK_24M, edp->regs + PLL_REG_1);
+
+ writel(0x95, edp->regs + PLL_REG_2);
+ writel(0x40, edp->regs + PLL_REG_3);
+ writel(0x58, edp->regs + PLL_REG_4);
+ writel(0x22, edp->regs + PLL_REG_5);
+ writel(0x19, edp->regs + SSC_REG);
+ writel(0x87, edp->regs + TX_REG_COMMON);
+ writel(0x03, edp->regs + DP_AUX);
+ writel(0x46, edp->regs + DP_BIAS);
+ writel(0x55, edp->regs + DP_RESERVE2);
+}
+
+void rockchip_edp_init_interrupt(struct rockchip_edp_device *edp)
+{
+ /* Set interrupt pin assertion polarity as high */
+ writel(INT_POL, edp->regs + INT_CTL);
+
+ /* Clear pending valisers */
+ writel(0xff, edp->regs + COMMON_INT_STA_1);
+ writel(0x4f, edp->regs + COMMON_INT_STA_2);
+ writel(0xff, edp->regs + COMMON_INT_STA_3);
+ writel(0x27, edp->regs + COMMON_INT_STA_4);
+
+ writel(0x7f, edp->regs + DP_INT_STA);
+
+ /* 0:mask,1: unmask */
+ writel(0x00, edp->regs + COMMON_INT_MASK_1);
+ writel(0x00, edp->regs + COMMON_INT_MASK_2);
+ writel(0x00, edp->regs + COMMON_INT_MASK_3);
+ writel(0x00, edp->regs + COMMON_INT_MASK_4);
+ writel(0x00, edp->regs + DP_INT_STA_MASK);
+}
+
+void rockchip_edp_reset(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ rockchip_edp_stop_video(edp);
+ rockchip_edp_enable_video_mute(edp, 0);
+
+ val = VID_CAP_FUNC_EN_N | AUD_FIFO_FUNC_EN_N |
+ AUD_FUNC_EN_N | HDCP_FUNC_EN_N | SW_FUNC_EN_N;
+ writel(val, edp->regs + FUNC_EN_1);
+
+ val = SSC_FUNC_EN_N | AUX_FUNC_EN_N |
+ SERDES_FIFO_FUNC_EN_N |
+ LS_CLK_DOMAIN_FUNC_EN_N;
+ writel(val, edp->regs + FUNC_EN_2);
+
+ usleep_range(20, 30);
+
+ rockchip_edp_lane_swap(edp, 0);
+
+ writel(0x0, edp->regs + SYS_CTL_1);
+ writel(0x40, edp->regs + SYS_CTL_2);
+ writel(0x0, edp->regs + SYS_CTL_3);
+ writel(0x0, edp->regs + SYS_CTL_4);
+
+ writel(0x0, edp->regs + PKT_SEND_CTL);
+ writel(0x0, edp->regs + HDCP_CTL);
+
+ writel(0x5e, edp->regs + HPD_DEGLITCH_L);
+ writel(0x1a, edp->regs + HPD_DEGLITCH_H);
+
+ writel(0x10, edp->regs + LINK_DEBUG_CTL);
+
+ writel(0x0, edp->regs + VIDEO_FIFO_THRD);
+ writel(0x20, edp->regs + AUDIO_MARGIN);
+
+ writel(0x4, edp->regs + M_VID_GEN_FILTER_TH);
+ writel(0x2, edp->regs + M_AUD_GEN_FILTER_TH);
+
+ writel(0x0, edp->regs + SOC_GENERAL_CTL);
+}
+
+void rockchip_edp_config_interrupt(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ /* 0: mask, 1: unmask */
+ val = 0;
+ writel(val, edp->regs + COMMON_INT_MASK_1);
+
+ writel(val, edp->regs + COMMON_INT_MASK_2);
+
+ writel(val, edp->regs + COMMON_INT_MASK_3);
+
+ writel(val, edp->regs + COMMON_INT_MASK_4);
+
+ writel(val, edp->regs + DP_INT_STA_MASK);
+}
+
+u32 rockchip_edp_get_pll_lock_status(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + DEBUG_CTL);
+
+ return (val & PLL_LOCK) ? DP_PLL_LOCKED : DP_PLL_UNLOCKED;
+}
+
+void rockchip_edp_analog_power_ctr(struct rockchip_edp_device *edp,
+ bool enable)
+{
+ u32 val;
+
+ if (enable) {
+ val = PD_EXP_BG | PD_AUX | PD_PLL |
+ PD_CH3 | PD_CH2 | PD_CH1 | PD_CH0;
+ writel(val, edp->regs + DP_PWRDN);
+ usleep_range(10, 20);
+ writel(0x0, edp->regs + DP_PWRDN);
+ } else {
+ val = PD_EXP_BG | PD_AUX | PD_PLL |
+ PD_CH3 | PD_CH2 | PD_CH1 | PD_CH0;
+ writel(val, edp->regs + DP_PWRDN);
+ }
+}
+
+void rockchip_edp_init_analog_func(struct rockchip_edp_device *edp)
+{
+ u32 val;
+ int wt = 0;
+
+ rockchip_edp_analog_power_ctr(edp, 1);
+
+ val = PLL_LOCK_CHG;
+ writel(val, edp->regs + COMMON_INT_STA_1);
+
+ val = readl(edp->regs + DEBUG_CTL);
+ val &= ~(F_PLL_LOCK | PLL_LOCK_CTRL);
+ writel(val, edp->regs + DEBUG_CTL);
+
+ /* Power up PLL */
+ while (wt < 100) {
+ if (rockchip_edp_get_pll_lock_status(edp) == DP_PLL_LOCKED) {
+ dev_dbg(edp->dev, "edp pll locked\n");
+ break;
+ }
+ wt++;
+ udelay(5);
+ }
+
+ /* Enable Serdes FIFO function and Link symbol clock domain module */
+ val = readl(edp->regs + FUNC_EN_2);
+ val &= ~(SERDES_FIFO_FUNC_EN_N | LS_CLK_DOMAIN_FUNC_EN_N
+ | AUX_FUNC_EN_N | SSC_FUNC_EN_N);
+ writel(val, edp->regs + FUNC_EN_2);
+}
+
+void rockchip_edp_init_hpd(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = HOTPLUG_CHG | HPD_LOST | PLUG;
+ writel(val, edp->regs + COMMON_INT_STA_4);
+
+ val = INT_HPD;
+ writel(val, edp->regs + DP_INT_STA);
+
+ val = readl(edp->regs + SYS_CTL_3);
+ val |= (F_HPD | HPD_CTRL);
+ writel(val, edp->regs + SYS_CTL_3);
+}
+
+void rockchip_edp_reset_aux(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ /* Disable AUX channel module */
+ val = readl(edp->regs + FUNC_EN_2);
+ val |= AUX_FUNC_EN_N;
+ writel(val, edp->regs + FUNC_EN_2);
+}
+
+void rockchip_edp_init_aux(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ /* Clear inerrupts related to AUX channel */
+ val = RPLY_RECEIV | AUX_ERR;
+ writel(val, edp->regs + DP_INT_STA);
+
+ rockchip_edp_reset_aux(edp);
+
+ /* Receive AUX Channel DEFER commands equal to DEFFER_COUNT*64 */
+ val = DEFER_CTRL_EN | DEFER_COUNT(1);
+ writel(val, edp->regs + AUX_CH_DEFER_CTL);
+
+ /* Enable AUX channel module */
+ val = readl(edp->regs + FUNC_EN_2);
+ val &= ~AUX_FUNC_EN_N;
+ writel(val, edp->regs + FUNC_EN_2);
+}
+
+int rockchip_edp_get_plug_in_status(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + SYS_CTL_3);
+ if (val & HPD_STATUS)
+ return 0;
+
+ return -EINVAL;
+}
+
+void rockchip_edp_enable_sw_function(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + FUNC_EN_1);
+ val &= ~SW_FUNC_EN_N;
+ writel(val, edp->regs + FUNC_EN_1);
+}
+
+int rockchip_edp_start_aux_transaction(struct rockchip_edp_device *edp)
+{
+ int val;
+ int retval = 0;
+ int timeout_loop = 0;
+ int aux_timeout = 0;
+
+ /* Enable AUX CH operation */
+ val = readl(edp->regs + AUX_CH_CTL_2);
+ val |= AUX_EN;
+ writel(val, edp->regs + AUX_CH_CTL_2);
+
+ /* Is AUX CH operation enabled? */
+ val = readl(edp->regs + AUX_CH_CTL_2);
+ while (val & AUX_EN) {
+ aux_timeout++;
+ if ((DP_TIMEOUT_LOOP_CNT * 10) < aux_timeout) {
+ dev_err(edp->dev, "AUX CH enable timeout!\n");
+ return -ETIMEDOUT;
+ }
+ val = readl(edp->regs + AUX_CH_CTL_2);
+ usleep_range(1000, 2000);
+ }
+
+ /* Is AUX CH command redply received? */
+ val = readl(edp->regs + DP_INT_STA);
+ while (!(val & RPLY_RECEIV)) {
+ timeout_loop++;
+ if (DP_TIMEOUT_LOOP_CNT < timeout_loop) {
+ dev_err(edp->dev, "AUX CH command redply failed!\n");
+ return -ETIMEDOUT;
+ }
+ val = readl(edp->regs + DP_INT_STA);
+ usleep_range(10, 20);
+ }
+
+ /* Clear interrupt source for AUX CH command redply */
+ writel(RPLY_RECEIV, edp->regs + DP_INT_STA);
+
+ /* Clear interrupt source for AUX CH access error */
+ val = readl(edp->regs + DP_INT_STA);
+ if (val & AUX_ERR) {
+ writel(AUX_ERR, edp->regs + DP_INT_STA);
+ return -EREMOTEIO;
+ }
+
+ /* Check AUX CH error access status */
+ val = readl(edp->regs + AUX_CH_STA);
+ if ((val & AUX_STATUS_MASK) != 0) {
+ dev_err(edp->dev, "AUX CH error happens: %d\n\n",
+ val & AUX_STATUS_MASK);
+ return -EREMOTEIO;
+ }
+
+ return retval;
+}
+
+int rockchip_edp_write_byte_to_dpcd(struct rockchip_edp_device *edp,
+ unsigned int val_addr,
+ unsigned char data)
+{
+ u32 val;
+ int i;
+ int retval;
+
+ for (i = 0; i < 3; i++) {
+ /* Clear AUX CH data buffer */
+ val = BUF_CLR;
+ writel(val, edp->regs + BUFFER_DATA_CTL);
+
+ /* Select DPCD device address */
+ val = AUX_ADDR_7_0(val_addr);
+ writel(val, edp->regs + DP_AUX_ADDR_7_0);
+ val = AUX_ADDR_15_8(val_addr);
+ writel(val, edp->regs + DP_AUX_ADDR_15_8);
+ val = AUX_ADDR_19_16(val_addr);
+ writel(val, edp->regs + DP_AUX_ADDR_19_16);
+
+ /* Write data buffer */
+ val = (unsigned int)data;
+ writel(val, edp->regs + BUF_DATA_0);
+
+ /*
+ * Set DisplayPort transaction and write 1 byte
+ * If bit 3 is 1, DisplayPort transaction.
+ * If Bit 3 is 0, I2C transaction.
+ */
+ val = AUX_TX_COMM_DP_TRANSACTION | AUX_TX_COMM_WRITE;
+ writel(val, edp->regs + AUX_CH_CTL_1);
+
+ /* Start AUX transaction */
+ retval = rockchip_edp_start_aux_transaction(edp);
+ if (retval == 0)
+ break;
+
+ dev_dbg(edp->dev, "Aux Transaction fail!\n");
+ }
+
+ return retval;
+}
+
+int rockchip_edp_read_byte_from_dpcd(struct rockchip_edp_device *edp,
+ unsigned int val_addr,
+ unsigned char *data)
+{
+ u32 val;
+ int i;
+ int retval;
+
+ for (i = 0; i < 10; i++) {
+ /* Clear AUX CH data buffer */
+ val = BUF_CLR;
+ writel(val, edp->regs + BUFFER_DATA_CTL);
+
+ /* Select DPCD device address */
+ val = AUX_ADDR_7_0(val_addr);
+ writel(val, edp->regs + DP_AUX_ADDR_7_0);
+ val = AUX_ADDR_15_8(val_addr);
+ writel(val, edp->regs + DP_AUX_ADDR_15_8);
+ val = AUX_ADDR_19_16(val_addr);
+ writel(val, edp->regs + DP_AUX_ADDR_19_16);
+
+ /*
+ * Set DisplayPort transaction and read 1 byte
+ * If bit 3 is 1, DisplayPort transaction.
+ * If Bit 3 is 0, I2C transaction.
+ */
+ val = AUX_TX_COMM_DP_TRANSACTION | AUX_TX_COMM_READ;
+ writel(val, edp->regs + AUX_CH_CTL_1);
+
+ /* Start AUX transaction */
+ retval = rockchip_edp_start_aux_transaction(edp);
+ if (retval == 0)
+ break;
+
+ dev_dbg(edp->dev, "Aux Transaction fail!\n");
+ }
+
+ /* Read data buffer */
+ val = readl(edp->regs + BUF_DATA_0);
+ *data = (unsigned char)(val & 0xff);
+
+ return retval;
+}
+
+int rockchip_edp_write_bytes_to_dpcd(struct rockchip_edp_device *edp,
+ unsigned int val_addr,
+ unsigned int count,
+ unsigned char data[])
+{
+ u32 val;
+ unsigned int start_offset;
+ unsigned int cur_data_count;
+ unsigned int cur_data_idx;
+ int i;
+ int retval = 0;
+
+ /* Clear AUX CH data buffer */
+ val = BUF_CLR;
+ writel(val, edp->regs + BUFFER_DATA_CTL);
+
+ start_offset = 0;
+ while (start_offset < count) {
+ /* Buffer size of AUX CH is 16 * 4bytes */
+ if ((count - start_offset) > 16)
+ cur_data_count = 16;
+ else
+ cur_data_count = count - start_offset;
+
+ for (i = 0; i < 10; i++) {
+ /* Select DPCD device address */
+ val = AUX_ADDR_7_0(val_addr + start_offset);
+ writel(val, edp->regs + DP_AUX_ADDR_7_0);
+ val = AUX_ADDR_15_8(val_addr + start_offset);
+ writel(val, edp->regs + DP_AUX_ADDR_15_8);
+ val = AUX_ADDR_19_16(val_addr + start_offset);
+ writel(val, edp->regs + DP_AUX_ADDR_19_16);
+
+ for (cur_data_idx = 0; cur_data_idx < cur_data_count;
+ cur_data_idx++) {
+ val = data[start_offset + cur_data_idx];
+ writel(val, edp->regs + BUF_DATA_0
+ + 4 * cur_data_idx);
+ }
+
+ /*
+ * Set DisplayPort transaction and write
+ * If bit 3 is 1, DisplayPort transaction.
+ * If Bit 3 is 0, I2C transaction.
+ */
+ val = AUX_LENGTH(cur_data_count) |
+ AUX_TX_COMM_DP_TRANSACTION | AUX_TX_COMM_WRITE;
+ writel(val, edp->regs + AUX_CH_CTL_1);
+
+ /* Start AUX transaction */
+ retval = rockchip_edp_start_aux_transaction(edp);
+ if (retval == 0)
+ break;
+
+ dev_dbg(edp->dev, "Aux Transaction fail!\n");
+ }
+
+ start_offset += cur_data_count;
+ }
+
+ return retval;
+}
+
+int rockchip_edp_read_bytes_from_dpcd(struct rockchip_edp_device *edp,
+ unsigned int val_addr,
+ unsigned int count,
+ unsigned char data[])
+{
+ u32 val;
+ unsigned int start_offset;
+ unsigned int cur_data_count;
+ unsigned int cur_data_idx;
+ int i;
+ int retval = 0;
+
+ /* Clear AUX CH data buffer */
+ val = BUF_CLR;
+ writel(val, edp->regs + BUFFER_DATA_CTL);
+
+ start_offset = 0;
+ while (start_offset < count) {
+ /* Buffer size of AUX CH is 16 * 4bytes */
+ if ((count - start_offset) > 16)
+ cur_data_count = 16;
+ else
+ cur_data_count = count - start_offset;
+
+ /* AUX CH Request Transaction process */
+ for (i = 0; i < 10; i++) {
+ /* Select DPCD device address */
+ val = AUX_ADDR_7_0(val_addr + start_offset);
+ writel(val, edp->regs + DP_AUX_ADDR_7_0);
+ val = AUX_ADDR_15_8(val_addr + start_offset);
+ writel(val, edp->regs + DP_AUX_ADDR_15_8);
+ val = AUX_ADDR_19_16(val_addr + start_offset);
+ writel(val, edp->regs + DP_AUX_ADDR_19_16);
+
+ /*
+ * Set DisplayPort transaction and read
+ * If bit 3 is 1, DisplayPort transaction.
+ * If Bit 3 is 0, I2C transaction.
+ */
+ val = AUX_LENGTH(cur_data_count) |
+ AUX_TX_COMM_DP_TRANSACTION | AUX_TX_COMM_READ;
+ writel(val, edp->regs + AUX_CH_CTL_1);
+
+ /* Start AUX transaction */
+ retval = rockchip_edp_start_aux_transaction(edp);
+ if (retval == 0)
+ break;
+
+ dev_dbg(edp->dev, "Aux Transaction fail!\n");
+ }
+
+ for (cur_data_idx = 0; cur_data_idx < cur_data_count;
+ cur_data_idx++) {
+ val = readl(edp->regs + BUF_DATA_0
+ + 4 * cur_data_idx);
+ data[start_offset + cur_data_idx] =
+ (unsigned char)val;
+ }
+
+ start_offset += cur_data_count;
+ }
+
+ return retval;
+}
+
+int rockchip_edp_select_i2c_device(struct rockchip_edp_device *edp,
+ unsigned int device_addr,
+ unsigned int val_addr)
+{
+ u32 val;
+ int retval;
+
+ /* Set EDID device address */
+ val = device_addr;
+ writel(val, edp->regs + DP_AUX_ADDR_7_0);
+ writel(0x0, edp->regs + DP_AUX_ADDR_15_8);
+ writel(0x0, edp->regs + DP_AUX_ADDR_19_16);
+
+ /* Set offset from base address of EDID device */
+ writel(val_addr, edp->regs + BUF_DATA_0);
+
+ /*
+ * Set I2C transaction and write address
+ * If bit 3 is 1, DisplayPort transaction.
+ * If Bit 3 is 0, I2C transaction.
+ */
+ val = AUX_TX_COMM_I2C_TRANSACTION | AUX_TX_COMM_MOT |
+ AUX_TX_COMM_WRITE;
+ writel(val, edp->regs + AUX_CH_CTL_1);
+
+ /* Start AUX transaction */
+ retval = rockchip_edp_start_aux_transaction(edp);
+ if (retval != 0)
+ dev_dbg(edp->dev, "Aux Transaction fail!\n");
+
+ return retval;
+}
+
+int rockchip_edp_read_byte_from_i2c(struct rockchip_edp_device *edp,
+ unsigned int device_addr,
+ unsigned int val_addr,
+ unsigned int *data)
+{
+ u32 val;
+ int i;
+ int retval;
+
+ for (i = 0; i < 10; i++) {
+ /* Clear AUX CH data buffer */
+ val = BUF_CLR;
+ writel(val, edp->regs + BUFFER_DATA_CTL);
+
+ /* Select EDID device */
+ retval = rockchip_edp_select_i2c_device(edp,
+ device_addr,
+ val_addr);
+ if (retval != 0) {
+ dev_err(edp->dev, "Select EDID device fail!\n");
+ continue;
+ }
+
+ /*
+ * Set I2C transaction and read data
+ * If bit 3 is 1, DisplayPort transaction.
+ * If Bit 3 is 0, I2C transaction.
+ */
+ val = AUX_TX_COMM_I2C_TRANSACTION | AUX_TX_COMM_READ;
+ writel(val, edp->regs + AUX_CH_CTL_1);
+
+ /* Start AUX transaction */
+ retval = rockchip_edp_start_aux_transaction(edp);
+ if (retval == 0)
+ break;
+
+ dev_dbg(edp->dev, "Aux Transaction fail!\n");
+ }
+
+ /* Read data */
+ if (retval == 0)
+ *data = readl(edp->regs + BUF_DATA_0);
+
+ return retval;
+}
+
+int rockchip_edp_read_bytes_from_i2c(struct rockchip_edp_device *edp,
+ unsigned int device_addr,
+ unsigned int val_addr,
+ unsigned int count,
+ unsigned char edid[])
+{
+ u32 val;
+ unsigned int i, j;
+ unsigned int cur_data_idx;
+ unsigned int defer = 0;
+ int retval = 0;
+
+ for (i = 0; i < count; i += 16) {
+ for (j = 0; j < 100; j++) {
+ /* Clear AUX CH data buffer */
+ val = BUF_CLR;
+ writel(val, edp->regs + BUFFER_DATA_CTL);
+
+ /* Set normal AUX CH command */
+ val = readl(edp->regs + AUX_CH_CTL_2);
+ val &= ~ADDR_ONLY;
+ writel(val, edp->regs + AUX_CH_CTL_2);
+
+ /*
+ * If Rx sends defer, Tx sends only reads
+ * request without sending addres
+ */
+ if (!defer)
+ retval = rockchip_edp_select_i2c_device(
+ edp, device_addr, val_addr + i);
+ else
+ defer = 0;
+
+ /*
+ * Set I2C transaction and write data
+ * If bit 3 is 1, DisplayPort transaction.
+ * If Bit 3 is 0, I2C transaction.
+ */
+ val = AUX_LENGTH(16) | AUX_TX_COMM_I2C_TRANSACTION |
+ AUX_TX_COMM_READ;
+ writel(val, edp->regs + AUX_CH_CTL_1);
+
+ /* Start AUX transaction */
+ retval = rockchip_edp_start_aux_transaction(edp);
+ if (retval == 0)
+ break;
+
+ dev_dbg(edp->dev, "Aux Transaction fail!\n");
+
+ /* Check if Rx sends defer */
+ val = readl(edp->regs + AUX_RX_COMM);
+ if (val == AUX_RX_COMM_AUX_DEFER ||
+ val == AUX_RX_COMM_I2C_DEFER) {
+ dev_err(edp->dev, "Defer: %d\n\n", val);
+ defer = 1;
+ }
+ }
+
+ for (cur_data_idx = 0; cur_data_idx < 16; cur_data_idx++) {
+ val = readl(edp->regs + BUF_DATA_0 + 4 * cur_data_idx);
+ edid[i + cur_data_idx] = (unsigned char)val;
+ }
+ }
+
+ return retval;
+}
+
+void rockchip_edp_set_link_bandwidth(struct rockchip_edp_device *edp,
+ u32 bwtype)
+{
+ u32 val;
+
+ val = bwtype;
+ if ((bwtype == DP_LINK_BW_2_7) || (bwtype == DP_LINK_BW_1_62))
+ writel(val, edp->regs + LINK_BW_SET);
+}
+
+void rockchip_edp_get_link_bandwidth(struct rockchip_edp_device *edp,
+ u32 *bwtype)
+{
+ u32 val;
+
+ val = readl(edp->regs + LINK_BW_SET);
+ *bwtype = val;
+}
+
+void rockchip_edp_hw_link_training_en(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = HW_LT_EN;
+ writel(val, edp->regs + HW_LT_CTL);
+}
+
+int rockchip_edp_wait_hw_lt_done(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + DP_INT_STA);
+ if (val&HW_LT_DONE) {
+ writel(val, edp->regs + DP_INT_STA);
+ return 0;
+ }
+
+ return 1;
+}
+
+int rockchip_edp_get_hw_lt_status(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + HW_LT_CTL);
+
+ return (val & HW_LT_ERR_CODE_MASK) >> 4;
+}
+
+void rockchip_edp_set_lane_count(struct rockchip_edp_device *edp, u32 count)
+{
+ u32 val;
+
+ val = count;
+ writel(val, edp->regs + LANE_CNT_SET);
+}
+
+void rockchip_edp_get_lane_count(struct rockchip_edp_device *edp, u32 *count)
+{
+ u32 val;
+
+ val = readl(edp->regs + LANE_CNT_SET);
+ *count = val;
+}
+
+void rockchip_edp_enable_enhanced_mode(struct rockchip_edp_device *edp,
+ bool enable)
+{
+ u32 val;
+
+ if (enable) {
+ val = readl(edp->regs + SYS_CTL_4);
+ val |= ENHANCED;
+ writel(val, edp->regs + SYS_CTL_4);
+ } else {
+ val = readl(edp->regs + SYS_CTL_4);
+ val &= ~ENHANCED;
+ writel(val, edp->regs + SYS_CTL_4);
+ }
+}
+
+void rockchip_edp_set_training_pattern(struct rockchip_edp_device *edp,
+ enum pattern_set pattern)
+{
+ u32 val;
+
+ switch (pattern) {
+ case PRBS7:
+ val = SCRAMBLING_ENABLE | LINK_QUAL_PATTERN_SET_PRBS7;
+ writel(val, edp->regs + TRAINING_PTN_SET);
+ break;
+ case D10_2:
+ val = SCRAMBLING_ENABLE | LINK_QUAL_PATTERN_SET_D10_2;
+ writel(val, edp->regs + TRAINING_PTN_SET);
+ break;
+ case TRAINING_PTN1:
+ val = SCRAMBLING_DISABLE | SW_TRAINING_PATTERN_SET_PTN1;
+ writel(val, edp->regs + TRAINING_PTN_SET);
+ break;
+ case TRAINING_PTN2:
+ val = SCRAMBLING_DISABLE | SW_TRAINING_PATTERN_SET_PTN2;
+ writel(val, edp->regs + TRAINING_PTN_SET);
+ break;
+ case DP_NONE:
+ val = SCRAMBLING_ENABLE |
+ LINK_QUAL_PATTERN_SET_DISABLE |
+ SW_TRAINING_PATTERN_SET_DISABLE;
+ writel(val, edp->regs + TRAINING_PTN_SET);
+ break;
+ default:
+ break;
+ }
+}
+
+void rockchip_edp_set_lane0_pre_emphasis(struct rockchip_edp_device *edp,
+ u32 level)
+{
+ u32 val;
+
+ val = level << PRE_EMPHASIS_SET_SHIFT;
+ writel(val, edp->regs + LN0_LINK_TRAINING_CTL);
+}
+
+void rockchip_edp_set_lane1_pre_emphasis(struct rockchip_edp_device *edp,
+ u32 level)
+{
+ u32 val;
+
+ val = level << PRE_EMPHASIS_SET_SHIFT;
+ writel(val, edp->regs + LN1_LINK_TRAINING_CTL);
+}
+
+void rockchip_edp_set_lane2_pre_emphasis(struct rockchip_edp_device *edp,
+ u32 level)
+{
+ u32 val;
+
+ val = level << PRE_EMPHASIS_SET_SHIFT;
+ writel(val, edp->regs + LN2_LINK_TRAINING_CTL);
+}
+
+void rockchip_edp_set_lane3_pre_emphasis(struct rockchip_edp_device *edp,
+ u32 level)
+{
+ u32 val;
+
+ val = level << PRE_EMPHASIS_SET_SHIFT;
+ writel(val, edp->regs + LN3_LINK_TRAINING_CTL);
+}
+
+void rockchip_edp_set_lane0_link_training(struct rockchip_edp_device *edp,
+ u32 training_lane)
+{
+ u32 val;
+
+ val = training_lane;
+ writel(val, edp->regs + LN0_LINK_TRAINING_CTL);
+}
+
+void rockchip_edp_set_lane1_link_training(struct rockchip_edp_device *edp,
+ u32 training_lane)
+{
+ u32 val;
+
+ val = training_lane;
+ writel(val, edp->regs + LN1_LINK_TRAINING_CTL);
+}
+
+void rockchip_edp_set_lane2_link_training(struct rockchip_edp_device *edp,
+ u32 training_lane)
+{
+ u32 val;
+
+ val = training_lane;
+ writel(val, edp->regs + LN2_LINK_TRAINING_CTL);
+}
+
+void rockchip_edp_set_lane3_link_training(struct rockchip_edp_device *edp,
+ u32 training_lane)
+{
+ u32 val;
+
+ val = training_lane;
+ writel(val, edp->regs + LN3_LINK_TRAINING_CTL);
+}
+
+u32 rockchip_edp_get_lane0_link_training(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + LN0_LINK_TRAINING_CTL);
+ return val;
+}
+
+u32 rockchip_edp_get_lane1_link_training(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + LN1_LINK_TRAINING_CTL);
+ return val;
+}
+
+u32 rockchip_edp_get_lane2_link_training(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + LN2_LINK_TRAINING_CTL);
+ return val;
+}
+
+u32 rockchip_edp_get_lane3_link_training(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + LN3_LINK_TRAINING_CTL);
+ return val;
+}
+
+void rockchip_edp_reset_macro(struct rockchip_edp_device *edp)
+{
+}
+
+int rockchip_edp_init_video(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = VSYNC_DET | VID_FORMAT_CHG | VID_CLK_CHG;
+ writel(val, edp->regs + COMMON_INT_STA_1);
+
+ val = 0x0;
+ writel(val, edp->regs + SYS_CTL_1);
+
+ val = CHA_CRI(4) | CHA_CTRL;
+ writel(val, edp->regs + SYS_CTL_2);
+
+ val = VID_HRES_TH(2) | VID_VRES_TH(0);
+ writel(val, edp->regs + VIDEO_CTL_8);
+
+ return 0;
+}
+
+void rockchip_edp_set_video_color_format(struct rockchip_edp_device *edp,
+ u32 color_dedpth,
+ u32 color_space,
+ u32 dynamic_range,
+ u32 coeff)
+{
+ u32 val;
+
+ /* Configure the input color dedpth, color space, dynamic range */
+ val = (dynamic_range << IN_D_RANGE_SHIFT) |
+ (color_dedpth << IN_BPC_SHIFT) |
+ (color_space << IN_COLOR_F_SHIFT);
+ writel(val, edp->regs + VIDEO_CTL_2);
+
+ /* Set Input Color YCbCr Coefficients to ITU601 or ITU709 */
+ val = readl(edp->regs + VIDEO_CTL_3);
+ val &= ~IN_YC_COEFFI_MASK;
+ if (coeff)
+ val |= IN_YC_COEFFI_ITU709;
+ else
+ val |= IN_YC_COEFFI_ITU601;
+ writel(val, edp->regs + VIDEO_CTL_3);
+}
+
+int rockchip_edp_is_slave_video_stream_clock_on(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + SYS_CTL_1);
+ writel(val, edp->regs + SYS_CTL_1);
+
+ val = readl(edp->regs + SYS_CTL_1);
+
+ if (!(val & DET_STA)) {
+ dev_dbg(edp->dev, "Input stream clock not detected.\n");
+ return -EINVAL;
+ }
+
+ val = readl(edp->regs + SYS_CTL_2);
+ writel(val, edp->regs + SYS_CTL_2);
+
+ val = readl(edp->regs + SYS_CTL_2);
+ if (val & CHA_STA) {
+ dev_dbg(edp->dev, "Input stream clk is changing\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+void rockchip_edp_set_video_cr_mn(struct rockchip_edp_device *edp,
+ enum clock_recovery_m_value_type type,
+ u32 m_value,
+ u32 n_value)
+{
+ u32 val;
+
+ if (type == REGISTER_M) {
+ val = readl(edp->regs + SYS_CTL_4);
+ val |= FIX_M_VID;
+ writel(val, edp->regs + SYS_CTL_4);
+ val = m_value & 0xff;
+ writel(val, edp->regs + M_VID_0);
+ val = (m_value >> 8) & 0xff;
+ writel(val, edp->regs + M_VID_1);
+ val = (m_value >> 16) & 0xff;
+ writel(val, edp->regs + M_VID_2);
+
+ val = n_value & 0xff;
+ writel(val, edp->regs + N_VID_0);
+ val = (n_value >> 8) & 0xff;
+ writel(val, edp->regs + N_VID_1);
+ val = (n_value >> 16) & 0xff;
+ writel(val, edp->regs + N_VID_2);
+ } else {
+ val = readl(edp->regs + SYS_CTL_4);
+ val &= ~FIX_M_VID;
+ writel(val, edp->regs + SYS_CTL_4);
+
+ writel(0x00, edp->regs + N_VID_0);
+ writel(0x80, edp->regs + N_VID_1);
+ writel(0x00, edp->regs + N_VID_2);
+ }
+}
+
+void rockchip_edp_set_video_timing_mode(struct rockchip_edp_device *edp,
+ u32 type)
+{
+ u32 val;
+
+ if (type == VIDEO_TIMING_FROM_CAPTURE) {
+ val = readl(edp->regs + VIDEO_CTL_10);
+ val &= ~F_SEL;
+ writel(val, edp->regs + VIDEO_CTL_10);
+ } else {
+ val = readl(edp->regs + VIDEO_CTL_10);
+ val |= F_SEL;
+ writel(val, edp->regs + VIDEO_CTL_10);
+ }
+}
+
+int rockchip_edp_bist_cfg(struct rockchip_edp_device *edp)
+{
+ struct video_info *video_info = &edp->video_info;
+ struct drm_display_mode *mode = &edp->mode;
+ u16 x_total, y_total, x_act;
+ u32 val;
+
+ x_total = mode->htotal;
+ y_total = mode->vtotal;
+ x_act = mode->hdisplay;
+
+ rockchip_edp_set_video_cr_mn(edp, CALCULATED_M, 0, 0);
+ rockchip_edp_set_video_color_format(edp, video_info->color_depth,
+ video_info->color_space,
+ video_info->dynamic_range,
+ video_info->ycbcr_coeff);
+
+ val = y_total & 0xff;
+ writel(val, edp->regs + TOTAL_LINE_CFG_L);
+ val = (y_total >> 8);
+ writel(val, edp->regs + TOTAL_LINE_CFG_H);
+ val = (mode->vdisplay & 0xff);
+ writel(val, edp->regs + ATV_LINE_CFG_L);
+ val = (mode->vdisplay >> 8);
+ writel(val, edp->regs + ATV_LINE_CFG_H);
+ val = (mode->vsync_start - mode->vdisplay);
+ writel(val, edp->regs + VF_PORCH_REG);
+ val = (mode->vsync_end - mode->vsync_start);
+ writel(val, edp->regs + VSYNC_CFG_REG);
+ val = (mode->vtotal - mode->vsync_end);
+ writel(val, edp->regs + VB_PORCH_REG);
+ val = x_total & 0xff;
+ writel(val, edp->regs + TOTAL_PIXELL_REG);
+ val = x_total >> 8;
+ writel(val, edp->regs + TOTAL_PIXELH_REG);
+ val = (x_act & 0xff);
+ writel(val, edp->regs + ATV_PIXELL_REG);
+ val = (x_act >> 8);
+ writel(val, edp->regs + ATV_PIXELH_REG);
+ val = (mode->hsync_start - mode->hdisplay) & 0xff;
+ writel(val, edp->regs + HF_PORCHL_REG);
+ val = (mode->hsync_start - mode->hdisplay) >> 8;
+ writel(val, edp->regs + HF_PORCHH_REG);
+ val = (mode->hsync_end - mode->hsync_start) & 0xff;
+ writel(val, edp->regs + HSYNC_CFGL_REG);
+ val = (mode->hsync_end - mode->hsync_start) >> 8;
+ writel(val, edp->regs + HSYNC_CFGH_REG);
+ val = (mode->htotal - mode->hsync_end) & 0xff;
+ writel(val, edp->regs + HB_PORCHL_REG);
+ val = (mode->htotal - mode->hsync_end) >> 8;
+ writel(val, edp->regs + HB_PORCHH_REG);
+
+ val = BIST_EN | BIST_WH_64 | BIST_TYPE_COLR_BAR;
+ writel(val, edp->regs + VIDEO_CTL_4);
+
+ val = readl(edp->regs + VIDEO_CTL_10);
+ val &= ~F_SEL;
+ writel(val, edp->regs + VIDEO_CTL_10);
+ return 0;
+}
+
+void rockchip_edp_enable_video_master(struct rockchip_edp_device *edp,
+ bool enable)
+{
+}
+
+void rockchip_edp_start_video(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + VIDEO_CTL_1);
+ val |= VIDEO_EN;
+ writel(val, edp->regs + VIDEO_CTL_1);
+}
+
+int rockchip_edp_is_video_stream_on(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + SYS_CTL_3);
+ writel(val, edp->regs + SYS_CTL_3);
+
+ val = readl(edp->regs + SYS_CTL_3);
+ if (!(val & STRM_VALID)) {
+ dev_dbg(edp->dev, "Input video stream is not detected.\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+void rockchip_edp_config_video_slave_mode(struct rockchip_edp_device *edp,
+ struct video_info *video_info)
+{
+ u32 val;
+
+ val = readl(edp->regs + FUNC_EN_1);
+ val &= ~(VID_FIFO_FUNC_EN_N | VID_CAP_FUNC_EN_N);
+ writel(val, edp->regs + FUNC_EN_1);
+
+ val = readl(edp->regs + VIDEO_CTL_10);
+ val &= ~INTERACE_SCAN_CFG;
+ val |= (video_info->interlaced << 2);
+ writel(val, edp->regs + VIDEO_CTL_10);
+
+ val = readl(edp->regs + VIDEO_CTL_10);
+ val &= ~VSYNC_POLARITY_CFG;
+ val |= (video_info->v_sync_polarity << 1);
+ writel(val, edp->regs + VIDEO_CTL_10);
+
+ val = readl(edp->regs + VIDEO_CTL_10);
+ val &= ~HSYNC_POLARITY_CFG;
+ val |= (video_info->h_sync_polarity << 0);
+ writel(val, edp->regs + VIDEO_CTL_10);
+}
+
+void rockchip_edp_enable_scrambling(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + TRAINING_PTN_SET);
+ val &= ~SCRAMBLING_DISABLE;
+ writel(val, edp->regs + TRAINING_PTN_SET);
+}
+
+void rockchip_edp_disable_scrambling(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = readl(edp->regs + TRAINING_PTN_SET);
+ val |= SCRAMBLING_DISABLE;
+ writel(val, edp->regs + TRAINING_PTN_SET);
+}
+
+enum dp_irq_type rockchip_edp_get_irq_type(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ /* Parse hotplug interrupt status register */
+ val = readl(edp->regs + COMMON_INT_STA_4);
+ if (val & PLUG)
+ return DP_IRQ_TYPE_HP_CABLE_IN;
+
+ if (val & HPD_LOST)
+ return DP_IRQ_TYPE_HP_CABLE_OUT;
+
+ if (val & HOTPLUG_CHG)
+ return DP_IRQ_TYPE_HP_CHANGE;
+
+ return DP_IRQ_TYPE_UNKNOWN;
+}
+
+void rockchip_edp_clear_hotplug_interrupts(struct rockchip_edp_device *edp)
+{
+ u32 val;
+
+ val = HOTPLUG_CHG | HPD_LOST | PLUG;
+ writel(val, edp->regs + COMMON_INT_STA_4);
+
+ val = INT_HPD;
+ writel(val, edp->regs + DP_INT_STA);
+}
diff --git a/drivers/gpu/drm/rockchip/rockchip_edp_reg.h b/drivers/gpu/drm/rockchip/rockchip_edp_reg.h
new file mode 100644
index 0000000..b50dd47
--- /dev/null
+++ b/drivers/gpu/drm/rockchip/rockchip_edp_reg.h
@@ -0,0 +1,345 @@
+/*
+* Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+* Author:
+* Andy yan <andy.yan-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
+* Jeff chen <jeff.chen-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
+*
+* based on exynos_dp_reg.h
+*
+* This program is free software; you can redistribute it and/or modify it
+* under the terms of the GNU General Public License as published by the
+* Free Software Foundation; either version 2 of the License, or (at your
+* option) any later version.
+*/
+
+#ifndef _ROCKCHIP_EDP_REG_H
+#define _ROCKCHIP_EDP_REG_H
+
+#include <linux/bitops.h>
+
+#define TX_SW_RST 0x14
+#define FUNC_EN_1 0x18
+#define FUNC_EN_2 0x1C
+#define VIDEO_CTL_1 0x20
+#define VIDEO_CTL_2 0x24
+#define VIDEO_CTL_3 0x28
+#define VIDEO_CTL_4 0x2c
+#define VIDEO_CTL_8 0x3C
+#define VIDEO_CTL_10 0x44
+#define TOTAL_LINE_CFG_L 0x48
+#define TOTAL_LINE_CFG_H 0x4c
+#define ATV_LINE_CFG_L 0x50
+#define ATV_LINE_CFG_H 0x54
+#define VF_PORCH_REG 0x58
+#define VSYNC_CFG_REG 0x5c
+#define VB_PORCH_REG 0x60
+#define TOTAL_PIXELL_REG 0x64
+#define TOTAL_PIXELH_REG 0x68
+#define ATV_PIXELL_REG 0x6c
+#define ATV_PIXELH_REG 0x70
+#define HF_PORCHL_REG 0x74
+#define HF_PORCHH_REG 0x78
+#define HSYNC_CFGL_REG 0x7c
+#define HSYNC_CFGH_REG 0x80
+#define HB_PORCHL_REG 0x84
+#define HB_PORCHH_REG 0x88
+#define PLL_REG_1 0xfc
+
+#define SSC_REG 0x104
+#define TX_REG_COMMON 0x114
+#define DP_AUX 0x120
+#define DP_BIAS 0x124
+#define DP_PWRDN 0x12c
+#define DP_RESERVE2 0x134
+
+#define LANE_MAP 0x35C
+#define ANALOG_CTL_2 0x374
+#define AUX_HW_RETRY_CTL 0x390
+#define COMMON_INT_STA_1 0x3C4
+#define COMMON_INT_STA_2 0x3C8
+#define COMMON_INT_STA_3 0x3CC
+#define COMMON_INT_STA_4 0x3D0
+#define DP_INT_STA 0x3DC
+#define COMMON_INT_MASK_1 0x3E0
+#define COMMON_INT_MASK_2 0x3E4
+#define COMMON_INT_MASK_3 0x3E8
+#define COMMON_INT_MASK_4 0x3EC
+#define DP_INT_STA_MASK 0x3F8
+
+#define SYS_CTL_1 0x600
+#define SYS_CTL_2 0x604
+#define SYS_CTL_3 0x608
+#define SYS_CTL_4 0x60C
+#define PKT_SEND_CTL 0x640
+#define HDCP_CTL 0x648
+#define LINK_BW_SET 0x680
+#define LANE_CNT_SET 0x684
+#define TRAINING_PTN_SET 0x688
+#define LN0_LINK_TRAINING_CTL 0x68C
+#define LN1_LINK_TRAINING_CTL 0x690
+#define LN2_LINK_TRAINING_CTL 0x694
+#define LN3_LINK_TRAINING_CTL 0x698
+#define HW_LT_CTL 0x6a0
+#define DEBUG_CTL 0x6C0
+#define HPD_DEGLITCH_L 0x6C4
+#define HPD_DEGLITCH_H 0x6C8
+#define LINK_DEBUG_CTL 0x6E0
+#define M_VID_0 0x700
+#define M_VID_1 0x704
+#define M_VID_2 0x708
+#define N_VID_0 0x70C
+#define N_VID_1 0x710
+#define N_VID_2 0x714
+#define VIDEO_FIFO_THRD 0x730
+#define AUDIO_MARGIN 0x73C
+#define M_VID_GEN_FILTER_TH 0x764
+#define M_AUD_GEN_FILTER_TH 0x778
+#define AUX_CH_STA 0x780
+#define AUX_CH_DEFER_CTL 0x788
+#define AUX_RX_COMM 0x78C
+#define BUFFER_DATA_CTL 0x790
+#define AUX_CH_CTL_1 0x794
+#define DP_AUX_ADDR_7_0 0x798
+#define DP_AUX_ADDR_15_8 0x79C
+#define DP_AUX_ADDR_19_16 0x7A0
+#define AUX_CH_CTL_2 0x7A4
+#define BUF_DATA_0 0x7C0
+#define SOC_GENERAL_CTL 0x800
+#define PLL_REG_2 0x9e4
+#define PLL_REG_3 0x9e8
+#define PLL_REG_4 0x9ec
+#define PLL_REG_5 0xa00
+
+/* ROCKCHIP_EDP_FUNC_EN_1 */
+#define VID_CAP_FUNC_EN_N BIT(6)
+#define VID_FIFO_FUNC_EN_N BIT(5)
+#define AUD_FIFO_FUNC_EN_N BIT(4)
+#define AUD_FUNC_EN_N BIT(3)
+#define HDCP_FUNC_EN_N BIT(2)
+#define SW_FUNC_EN_N BIT(0)
+
+/* ROCKCHIP_EDP_FUNC_EN_2 */
+#define SSC_FUNC_EN_N BIT(7)
+#define AUX_FUNC_EN_N BIT(2)
+#define SERDES_FIFO_FUNC_EN_N BIT(1)
+#define LS_CLK_DOMAIN_FUNC_EN_N BIT(0)
+
+/* ROCKCHIP_EDP_VIDEO_CTL_1 */
+#define VIDEO_EN BIT(7)
+#define VIDEO_MUTE BIT(6)
+
+/* ROCKCHIP_EDP_VIDEO_CTL_1 */
+#define IN_D_RANGE_MASK (0x1 << 7)
+#define IN_D_RANGE_SHIFT (7)
+#define IN_D_RANGE_CEA (0x1 << 7)
+#define IN_D_RANGE_VESA (0x0 << 7)
+#define IN_BPC_MASK (0x7 << 4)
+#define IN_BPC_SHIFT (4)
+#define IN_BPC_12_BITS (0x3 << 4)
+#define IN_BPC_10_BITS (0x2 << 4)
+#define IN_BPC_8_BITS (0x1 << 4)
+#define IN_BPC_6_BITS (0x0 << 4)
+#define IN_COLOR_F_MASK (0x3 << 0)
+#define IN_COLOR_F_SHIFT (0)
+#define IN_COLOR_F_YCBCR444 (0x2 << 0)
+#define IN_COLOR_F_YCBCR422 (0x1 << 0)
+#define IN_COLOR_F_RGB (0x0 << 0)
+
+/* ROCKCHIP_EDP_VIDEO_CTL_3 */
+#define IN_YC_COEFFI_MASK (0x1 << 7)
+#define IN_YC_COEFFI_SHIFT (7)
+#define IN_YC_COEFFI_ITU709 (0x1 << 7)
+#define IN_YC_COEFFI_ITU601 (0x0 << 7)
+#define VID_CHK_UPDATE_TYPE_MASK (0x1 << 4)
+#define VID_CHK_UPDATE_TYPE_SHIFT (4)
+#define VID_CHK_UPDATE_TYPE_1 (0x1 << 4)
+#define VID_CHK_UPDATE_TYPE_0 (0x0 << 4)
+
+/* ROCKCHIP_EDP_VIDEO_CTL_4 */
+#define BIST_EN (0x1 << 3)
+#define BIST_WH_64 (0x1 << 2)
+#define BIST_WH_32 (0x0 << 2)
+#define BIST_TYPE_COLR_BAR (0x0 << 0)
+#define BIST_TYPE_GRAY_BAR (0x1 << 0)
+#define BIST_TYPE_MOBILE_BAR (0x2 << 0)
+
+/* ROCKCHIP_EDP_VIDEO_CTL_8 */
+#define VID_HRES_TH(x) (((x) & 0xf) << 4)
+#define VID_VRES_TH(x) (((x) & 0xf) << 0)
+
+/* ROCKCHIP_EDP_VIDEO_CTL_10 */
+#define F_SEL (0x1 << 4)
+#define INTERACE_SCAN_CFG (0x1 << 2)
+#define VSYNC_POLARITY_CFG (0x1 << 1)
+#define HSYNC_POLARITY_CFG (0x1 << 0)
+
+/* ROCKCHIP_EDP_PLL_REG_1 */
+#define REF_CLK_24M (0x1 << 1)
+#define REF_CLK_27M (0x0 << 1)
+
+/* ROCKCHIP_EDP_DP_PWRDN */
+#define PD_INC_BG BIT(7)
+#define PD_EXP_BG BIT(6)
+#define PD_AUX BIT(5)
+#define PD_PLL BIT(4)
+#define PD_CH3 BIT(3)
+#define PD_CH2 BIT(2)
+#define PD_CH1 BIT(1)
+#define PD_CH0 BIT(0)
+
+/* ROCKCHIP_EDP_LANE_MAP */
+#define LANE3_MAP_LOGIC_LANE_0 (0x0 << 6)
+#define LANE3_MAP_LOGIC_LANE_1 (0x1 << 6)
+#define LANE3_MAP_LOGIC_LANE_2 (0x2 << 6)
+#define LANE3_MAP_LOGIC_LANE_3 (0x3 << 6)
+#define LANE2_MAP_LOGIC_LANE_0 (0x0 << 4)
+#define LANE2_MAP_LOGIC_LANE_1 (0x1 << 4)
+#define LANE2_MAP_LOGIC_LANE_2 (0x2 << 4)
+#define LANE2_MAP_LOGIC_LANE_3 (0x3 << 4)
+#define LANE1_MAP_LOGIC_LANE_0 (0x0 << 2)
+#define LANE1_MAP_LOGIC_LANE_1 (0x1 << 2)
+#define LANE1_MAP_LOGIC_LANE_2 (0x2 << 2)
+#define LANE1_MAP_LOGIC_LANE_3 (0x3 << 2)
+#define LANE0_MAP_LOGIC_LANE_0 (0x0 << 0)
+#define LANE0_MAP_LOGIC_LANE_1 (0x1 << 0)
+#define LANE0_MAP_LOGIC_LANE_2 (0x2 << 0)
+#define LANE0_MAP_LOGIC_LANE_3 (0x3 << 0)
+
+/* ROCKCHIP_EDP_ANALOG_CTL_2 */
+#define SEL_24M (0x1 << 3)
+
+/* ROCKCHIP_EDP_COMMON_INT_STA_1 */
+#define VSYNC_DET BIT(7)
+#define PLL_LOCK_CHG BIT(6)
+#define SPDIF_ERR BIT(5)
+#define SPDIF_UNSTBL BIT(4)
+#define VID_FORMAT_CHG BIT(3)
+#define AUD_CLK_CHG BIT(2)
+#define VID_CLK_CHG BIT(1)
+#define SW_INT BIT(0)
+
+/* ROCKCHIP_EDP_COMMON_INT_STA_2 */
+#define ENC_EN_CHG BIT(6)
+#define HW_BKSV_RDY BIT(3)
+#define HW_SHA_DONE BIT(2)
+#define HW_AUTH_STATE_CHG BIT(1)
+#define HW_AUTH_DONE BIT(0)
+
+/* ROCKCHIP_EDP_COMMON_INT_STA_3 */
+#define AFIFO_UNDER BIT(7)
+#define AFIFO_OVER BIT(6)
+#define R0_CHK_FLAG BIT(5)
+
+/* ROCKCHIP_EDP_COMMON_INT_STA_4 */
+#define PSR_ACTIVE BIT(7)
+#define PSR_INACTIVE BIT(6)
+#define SPDIF_BI_PHASE_ERR BIT(5)
+#define HOTPLUG_CHG BIT(2)
+#define HPD_LOST BIT(1)
+#define PLUG BIT(0)
+
+/* ROCKCHIP_EDP_INT_STA */
+#define INT_HPD BIT(6)
+#define HW_LT_DONE BIT(5)
+#define SINK_LOST BIT(3)
+#define LINK_LOST BIT(2)
+#define RPLY_RECEIV BIT(1)
+#define AUX_ERR BIT(0)
+
+/* ROCKCHIP_EDP_INT_CTL */
+#define INT_CTL 0x3FC
+#define SOFT_INT_CTRL BIT(2)
+#define INT_POL BIT(0)
+
+/* ROCKCHIP_EDP_SYS_CTL_1 */
+#define DET_STA BIT(2)
+#define FORCE_DET BIT(1)
+#define DET_CTRL BIT(0)
+
+/* ROCKCHIP_EDP_SYS_CTL_2 */
+#define CHA_CRI(x) (((x) & 0xf) << 4)
+#define CHA_STA BIT(2)
+#define FORCE_CHA BIT(1)
+#define CHA_CTRL BIT(0)
+
+/* ROCKCHIP_EDP_SYS_CTL_3 */
+#define HPD_STATUS BIT(6)
+#define F_HPD BIT(5)
+#define HPD_CTRL BIT(4)
+#define HDCP_RDY BIT(3)
+#define STRM_VALID BIT(2)
+#define F_VALID BIT(1)
+#define VALID_CTRL BIT(0)
+
+/* ROCKCHIP_EDP_SYS_CTL_4 */
+#define FIX_M_AUD BIT(4)
+#define ENHANCED BIT(3)
+#define FIX_M_VID BIT(2)
+#define M_VID_UPDATE_CTRL BIT(0)
+
+/* ROCKCHIP_EDP_TRAINING_PTN_SET */
+#define SCRAMBLING_DISABLE (0x1 << 5)
+#define SCRAMBLING_ENABLE (0x0 << 5)
+#define LINK_QUAL_PATTERN_SET_MASK (0x7 << 2)
+#define LINK_QUAL_PATTERN_SET_PRBS7 (0x3 << 2)
+#define LINK_QUAL_PATTERN_SET_D10_2 (0x1 << 2)
+#define LINK_QUAL_PATTERN_SET_DISABLE (0x0 << 2)
+#define SW_TRAINING_PATTERN_SET_MASK (0x3 << 0)
+#define SW_TRAINING_PATTERN_SET_PTN2 (0x2 << 0)
+#define SW_TRAINING_PATTERN_SET_PTN1 (0x1 << 0)
+#define SW_TRAINING_PATTERN_SET_DISABLE (0x0 << 0)
+
+/* ROCKCHIP_EDP_HW_LT_CTL */
+#define HW_LT_ERR_CODE_MASK 0x70
+#define HW_LT_EN BIT(0)
+
+/* ROCKCHIP_EDP_LN0_LINK_TRAINING_CTL */
+#define PRE_EMPHASIS_SET_MASK (0x3 << 3)
+#define PRE_EMPHASIS_SET_SHIFT (3)
+
+/* ROCKCHIP_EDP_DEBUG_CTL */
+#define PLL_LOCK BIT(4)
+#define F_PLL_LOCK BIT(3)
+#define PLL_LOCK_CTRL BIT(2)
+#define POLL_EN BIT(1)
+#define PN_INV BIT(0)
+
+/* ROCKCHIP_EDP_AUX_CH_STA */
+#define AUX_BUSY (0x1 << 4)
+#define AUX_STATUS_MASK (0xf << 0)
+
+/* ROCKCHIP_EDP_AUX_CH_DEFER_CTL */
+#define DEFER_CTRL_EN (0x1 << 7)
+#define DEFER_COUNT(x) (((x) & 0x7f) << 0)
+
+/* ROCKCHIP_EDP_AUX_RX_COMM */
+#define AUX_RX_COMM_I2C_DEFER (0x2 << 2)
+#define AUX_RX_COMM_AUX_DEFER (0x2 << 0)
+
+/* ROCKCHIP_EDP_BUFFER_DATA_CTL */
+#define BUF_CLR (0x1 << 7)
+#define BUF_DATA_COUNT(x) (((x) & 0xf) << 0)
+
+/* ROCKCHIP_EDP_AUX_CH_CTL_1 */
+#define AUX_LENGTH(x) (((x - 1) & 0xf) << 4)
+#define AUX_TX_COMM_MASK (0xf << 0)
+#define AUX_TX_COMM_DP_TRANSACTION (0x1 << 3)
+#define AUX_TX_COMM_I2C_TRANSACTION (0x0 << 3)
+#define AUX_TX_COMM_MOT (0x1 << 2)
+#define AUX_TX_COMM_WRITE (0x0 << 0)
+#define AUX_TX_COMM_READ (0x1 << 0)
+
+/* OCKCHIP_EDP_AUX_ADDR_7_0 */
+#define AUX_ADDR_7_0(x) (((x) >> 0) & 0xff)
+
+/* ROCKCHIP_EDP_AUX_ADDR_15_8 */
+#define AUX_ADDR_15_8(x) (((x) >> 8) & 0xff)
+
+/* ROCKCHIP_EDP_AUX_ADDR_19_16 */
+#define AUX_ADDR_19_16(x) (((x) >> 16) & 0x0f)
+
+/* ROCKCHIP_EDP_AUX_CH_CTL_2 */
+#define ADDR_ONLY BIT(1)
+#define AUX_EN BIT(0)
+
+#endif /* _ROCKCHIP_EDP_REG_H */
--
1.7.9.5
^ permalink raw reply related
* Re: [PATCH v4 1/5] drm/rockchip: Add basic drm driver
From: Boris BREZILLON @ 2014-09-22 13:24 UTC (permalink / raw)
To: Mark yao
Cc: heiko-4mtYJXux2i+zQB+pC5nmwQ, David Airlie, Rob Clark,
Rob Herring, Pawel Moll, Mark Rutland, Ian Campbell, Kumar Gala,
Randy Dunlap, Grant Likely, Greg Kroah-Hartman, John Stultz,
Rom Lemarchand, devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-doc-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
dianders-F7+t8E8rja9g9hUCZPvPmw, marcheu-F7+t8E8rja9g9hUCZPvPmw,
dbehr-F7+t8E8rja9g9hUCZPvPmw, olof-nZhT3qVonbNeoWH0uzbU5w,
djkurtz-F7+t8E8rja9g9hUCZPvPmw, xjq-TNX95d0MmH7DzftRWevZcw,
kfx-TNX95d0MmH7DzftRWevZcw, cym-TNX95d0MmH7DzftRWevZcw,
cf-TNX95d0MmH7DzftRWevZcw, zyw-TNX95d0MmH7DzftRWevZcw,
xxm-TNX95d0MmH7DzftRWevZcw, huang
In-Reply-To: <1411382934-1763-1-git-send-email-mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
Hi Mark,
You'll find some comments inline.
Anyway, I wouldn't call it a review (your driver is using some concepts
I'm not used to, like IOMMUs) but rather a collection of nitpicks :-).
I haven't been through the whole driver yet, but I'll get back to it
soon ;-).
And remember this is a 2 way thing, I wait for your review too
(here is the last version of my driver [1]) :-)
On Mon, 22 Sep 2014 18:48:54 +0800
Mark yao <mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org> wrote:
> This patch adds the basic structure of a DRM Driver for Rockchip Socs.
>
> Signed-off-by: Mark yao <mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
> ---
> Changes in v2:
> - use the component framework to defer main drm driver probe
> until all VOP devices have been probed.
> - use dma-mapping API with ARM_DMA_USE_IOMMU, create dma mapping by
> master device and each vop device can shared the drm dma mapping.
> - use drm_crtc_init_with_planes and drm_universal_plane_init.
> - remove unnecessary middle layers.
> - add cursor set, move funcs to rockchip drm crtc.
> - use vop reset at first init
> - reference framebuffer when used and unreference when swap out vop
>
> Changes in v3:
> - change "crtc->fb" to "crtc->primary-fb"
> Adviced by Daniel Vetter
> - init cursor plane with universal api, remove unnecessary cursor set,move
>
> Changes in v4:
> Adviced by David Herrmann
> - remove drm_platform_*() usage, use register drm device directly.
> Adviced by Rob Clark
> - remove special mmap ioctl, do userspace mmap with normal mmap() or mmap offset
>
> drivers/gpu/drm/Kconfig | 2 +
> drivers/gpu/drm/Makefile | 1 +
> drivers/gpu/drm/rockchip/Kconfig | 19 +
> drivers/gpu/drm/rockchip/Makefile | 10 +
> drivers/gpu/drm/rockchip/rockchip_drm_drv.c | 524 ++++++++++
> drivers/gpu/drm/rockchip/rockchip_drm_drv.h | 120 +++
> drivers/gpu/drm/rockchip/rockchip_drm_fb.c | 201 ++++
> drivers/gpu/drm/rockchip/rockchip_drm_fb.h | 28 +
> drivers/gpu/drm/rockchip/rockchip_drm_fbdev.c | 231 +++++
> drivers/gpu/drm/rockchip/rockchip_drm_fbdev.h | 20 +
> drivers/gpu/drm/rockchip/rockchip_drm_gem.c | 404 ++++++++
> drivers/gpu/drm/rockchip/rockchip_drm_gem.h | 72 ++
> drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 1372 +++++++++++++++++++++++++
> drivers/gpu/drm/rockchip/rockchip_drm_vop.h | 187 ++++
> include/uapi/drm/rockchip_drm.h | 75 ++
> 15 files changed, 3266 insertions(+)
> create mode 100644 drivers/gpu/drm/rockchip/Kconfig
> create mode 100644 drivers/gpu/drm/rockchip/Makefile
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_drv.c
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_drv.h
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_fb.c
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_fb.h
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_fbdev.c
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_fbdev.h
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_gem.c
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_gem.h
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_vop.c
> create mode 100644 drivers/gpu/drm/rockchip/rockchip_drm_vop.h
> create mode 100644 include/uapi/drm/rockchip_drm.h
>
> diff --git a/drivers/gpu/drm/Kconfig b/drivers/gpu/drm/Kconfig
> index b066bb3..7c4c3c6 100644
> --- a/drivers/gpu/drm/Kconfig
> +++ b/drivers/gpu/drm/Kconfig
> @@ -171,6 +171,8 @@ config DRM_SAVAGE
>
> source "drivers/gpu/drm/exynos/Kconfig"
>
> +source "drivers/gpu/drm/rockchip/Kconfig"
> +
> source "drivers/gpu/drm/vmwgfx/Kconfig"
>
> source "drivers/gpu/drm/gma500/Kconfig"
> diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
> index 4a55d59..d03387a 100644
> --- a/drivers/gpu/drm/Makefile
> +++ b/drivers/gpu/drm/Makefile
> @@ -52,6 +52,7 @@ obj-$(CONFIG_DRM_VMWGFX)+= vmwgfx/
> obj-$(CONFIG_DRM_VIA) +=via/
> obj-$(CONFIG_DRM_NOUVEAU) +=nouveau/
> obj-$(CONFIG_DRM_EXYNOS) +=exynos/
> +obj-$(CONFIG_DRM_ROCKCHIP) +=rockchip/
> obj-$(CONFIG_DRM_GMA500) += gma500/
> obj-$(CONFIG_DRM_UDL) += udl/
> obj-$(CONFIG_DRM_AST) += ast/
> diff --git a/drivers/gpu/drm/rockchip/Kconfig b/drivers/gpu/drm/rockchip/Kconfig
> new file mode 100644
> index 0000000..7146c80
> --- /dev/null
> +++ b/drivers/gpu/drm/rockchip/Kconfig
> @@ -0,0 +1,19 @@
> +config DRM_ROCKCHIP
> + tristate "DRM Support for Rockchip"
> + depends on DRM && ROCKCHIP_IOMMU
> + select ARM_DMA_USE_IOMMU
> + select IOMMU_API
> + select DRM_KMS_HELPER
> + select DRM_KMS_FB_HELPER
> + select DRM_PANEL
> + select FB_CFB_FILLRECT
> + select FB_CFB_COPYAREA
> + select FB_CFB_IMAGEBLIT
> + select VT_HW_CONSOLE_BINDING if FRAMEBUFFER_CONSOLE
> + select VIDEOMODE_HELPERS
> + help
> + Choose this option if you have a Rockchip soc chipset.
> + This driver provides kernel mode setting and buffer
> + management to userspace. This driver does not provides
> + 2D or 3D acceleration; acceleration is performed by other
> + IP found on the SoC.
> diff --git a/drivers/gpu/drm/rockchip/Makefile b/drivers/gpu/drm/rockchip/Makefile
> new file mode 100644
> index 0000000..6e6d468
> --- /dev/null
> +++ b/drivers/gpu/drm/rockchip/Makefile
> @@ -0,0 +1,10 @@
> +#
> +# Makefile for the drm device driver. This driver provides support for the
> +# Direct Rendering Infrastructure (DRI) in XFree86 4.1.0 and higher.
> +
> +ccflags-y := -Iinclude/drm -Idrivers/gpu/drm/rockchip
Do you really need those specific CFLAGS (AFAIK, these path are
already set, though you'll have to include <drm/xxx.h> instead of
"xxx.h" if you're referencing drm headers, but you're already
referencing the correct patch anyway) ?
> +
> +rockchipdrm-y := rockchip_drm_drv.o rockchip_drm_fb.o rockchip_drm_fbdev.o \
> + rockchip_drm_gem.o rockchip_drm_vop.o
> +
> +obj-$(CONFIG_DRM_ROCKCHIP) += rockchipdrm.o
> diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_drv.c b/drivers/gpu/drm/rockchip/rockchip_drm_drv.c
> new file mode 100644
> index 0000000..94926cb
> --- /dev/null
> +++ b/drivers/gpu/drm/rockchip/rockchip_drm_drv.c
> @@ -0,0 +1,524 @@
> +/*
> + * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
> + * Author:Mark Yao <mark.yao-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
> + *
> + * based on exynos_drm_drv.c
> + *
> + * This software is licensed under the terms of the GNU General Public
> + * License version 2, as published by the Free Software Foundation, and
> + * may be copied, distributed, and modified under those terms.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
[...]
> +
> +static int rockchip_drm_load(struct drm_device *drm_dev, unsigned long flags)
> +{
> + struct rockchip_drm_private *private;
> + struct dma_iommu_mapping *mapping;
> + struct device *dev = drm_dev->dev;
> + int ret;
> +
> + private = devm_kzalloc(drm_dev->dev, sizeof(*private), GFP_KERNEL);
> + if (!private)
> + return -ENOMEM;
> +
> + dev_set_drvdata(drm_dev->dev, dev);
> + drm_dev->dev_private = private;
> +
> + drm_mode_config_init(drm_dev);
> +
> + rockchip_drm_mode_config_init(drm_dev);
> +
> + dev->dma_parms = devm_kzalloc(dev, sizeof(*dev->dma_parms),
> + GFP_KERNEL);
> + if (!dev->dma_parms) {
> + ret = -ENOMEM;
> + goto err_config_cleanup;
> + }
> +
> + /* TODO(djkurtz): fetch the mapping start/size from somewhere */
> + mapping = arm_iommu_create_mapping(&platform_bus_type, 0x10000000,
> + SZ_1G);
> + if (IS_ERR(mapping)) {
> + ret = PTR_ERR(mapping);
> + goto err_config_cleanup;
> + }
> +
> + dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
> + dma_set_max_seg_size(dev, 0xffffffffu);
> +
> + ret = arm_iommu_attach_device(dev, mapping);
> + if (ret)
> + goto err_release_mapping;
> +
> + /* Try to bind all sub drivers. */
> + ret = component_bind_all(dev, drm_dev);
> + if (ret)
> + goto err_detach_device;
> +
> + /* init kms poll for handling hpd */
> + drm_kms_helper_poll_init(drm_dev);
> +
> + /*
> + * enable drm irq mode.
> + * - with irq_enabled = true, we can use the vblank feature.
> + */
> + drm_dev->irq_enabled = true;
> +
> + /*
> + * with vblank_disable_allowed = true, vblank interrupt will be disabled
> + * by drm timer once a current process gives up ownership of
> + * vblank event.(after drm_vblank_put function is called)
> + */
> + drm_dev->vblank_disable_allowed = true;
> +
> + ret = drm_vblank_init(drm_dev, ROCKCHIP_MAX_CRTC);
> + if (ret)
> + goto err_kms_helper_poll_fini;
> +
> + rockchip_drm_fbdev_init(drm_dev);
> +
> + /* force connectors detection */
> + drm_helper_hpd_irq_event(drm_dev);
> +
> + return 0;
> +
> +err_kms_helper_poll_fini:
> + drm_kms_helper_poll_fini(drm_dev);
> + component_unbind_all(dev, drm_dev);
> +err_detach_device:
> + arm_iommu_detach_device(dev);
> +err_release_mapping:
> + arm_iommu_release_mapping(dev->archdata.mapping);
> +err_config_cleanup:
> + drm_mode_config_cleanup(drm_dev);
> + drm_dev->dev_private = NULL;
> + dev_set_drvdata(dev, NULL);
Not sure you need to set driver data to NULL.
> + return ret;
> +}
> +
[...]
> +
> +#ifdef CONFIG_PM_SLEEP
> +static int rockchip_drm_sys_suspend(struct device *dev)
> +{
> + struct drm_device *drm_dev = dev_get_drvdata(dev);
> + pm_message_t message;
> +
> + if (pm_runtime_suspended(dev))
> + return 0;
> +
> + message.event = PM_EVENT_SUSPEND;
> +
> + return rockchip_drm_suspend(drm_dev, message);
> +}
> +
> +static int rockchip_drm_sys_resume(struct device *dev)
> +{
> + struct drm_device *drm_dev = dev_get_drvdata(dev);
> +
> + if (pm_runtime_suspended(dev))
You meant
if (!pm_runtime_suspended(dev))
right ?
BTW, I see the same mistake in exynos driver [2]
> + return 0;
> +
> + return rockchip_drm_resume(drm_dev);
> +}
> +#endif
> +
> +static const struct dev_pm_ops rockchip_drm_pm_ops = {
> + SET_SYSTEM_SLEEP_PM_OPS(rockchip_drm_sys_suspend,
> + rockchip_drm_sys_resume)
> +};
> +
> +int rockchip_drm_add_crtc(struct drm_device *drm, struct drm_crtc *crtc,
> + struct device_node *np)
> +{
> + struct rockchip_drm_private *priv = drm->dev_private;
> + struct device_node *port;
> + int pipe;
> +
> + if (priv->num_pipe >= ROCKCHIP_MAX_CRTC)
> + return -EINVAL;
> +
> + port = of_get_child_by_name(np, "port");
> + of_node_put(np);
Not sure you should call of_node_put on a node pointer passed as an
argument (unless you previously called of_node_get which is not the case
in this function)...
Best Regards,
Boris
[1]http://thread.gmane.org/gmane.comp.video.dri.devel/114064
[2]https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/tree/drivers/gpu/drm/exynos/exynos_drm_drv.c?id=refs/tags/next-20140922#n373
--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply
* Re: [PATCH v4 1/5] drm/rockchip: Add basic drm driver
From: Arnd Bergmann @ 2014-09-22 14:43 UTC (permalink / raw)
To: Mark yao
Cc: heiko, Boris BREZILLON, David Airlie, Rob Clark, Rob Herring,
Pawel Moll, Mark Rutland, Ian Campbell, Kumar Gala, Randy Dunlap,
Grant Likely, Greg Kroah-Hartman, John Stultz, Rom Lemarchand,
devicetree, linux-doc, linux-kernel, dri-devel, linux-api,
linux-rockchip, dianders, marcheu, dbehr, olof, djkurtz, xjq, kfx,
cym, cf
In-Reply-To: <1411382934-1763-1-git-send-email-mark.yao@rock-chips.com>
On Monday 22 September 2014 18:48:54 Mark yao wrote:
> diff --git a/drivers/gpu/drm/rockchip/Kconfig b/drivers/gpu/drm/rockchip/Kconfig
> new file mode 100644
> index 0000000..7146c80
> --- /dev/null
> +++ b/drivers/gpu/drm/rockchip/Kconfig
> @@ -0,0 +1,19 @@
> +config DRM_ROCKCHIP
> + tristate "DRM Support for Rockchip"
> + depends on DRM && ROCKCHIP_IOMMU
> + select ARM_DMA_USE_IOMMU
> + select IOMMU_API
> + select DRM_KMS_HELPER
> + select DRM_KMS_FB_HELPER
> + select DRM_PANEL
> + select FB_CFB_FILLRECT
> + select FB_CFB_COPYAREA
> + select FB_CFB_IMAGEBLIT
> + select VT_HW_CONSOLE_BINDING if FRAMEBUFFER_CONSOLE
> + select VIDEOMODE_HELPERS
Be careful with 'select', at least some of these should be 'depends on'.
In particular IOMMU_API and ARM_DMA_USE_IOMMU, but possibly others
as well. Just check how the symbols are used normally, if you get this
wrong, we can end up with incorrect dependencies or loops.
> diff --git a/drivers/gpu/drm/rockchip/Makefile b/drivers/gpu/drm/rockchip/Makefile
> new file mode 100644
> index 0000000..6e6d468
> --- /dev/null
> +++ b/drivers/gpu/drm/rockchip/Makefile
> @@ -0,0 +1,10 @@
> +#
> +# Makefile for the drm device driver. This driver provides support for the
> +# Direct Rendering Infrastructure (DRI) in XFree86 4.1.0 and higher.
> +
> +ccflags-y := -Iinclude/drm -Idrivers/gpu/drm/rockchip
The second one should not be required.
> +static int rockchip_drm_load(struct drm_device *drm_dev, unsigned long flags)
> +{
> + struct rockchip_drm_private *private;
> + struct dma_iommu_mapping *mapping;
> + struct device *dev = drm_dev->dev;
> + int ret;
> +
> + private = devm_kzalloc(drm_dev->dev, sizeof(*private), GFP_KERNEL);
> + if (!private)
> + return -ENOMEM;
> +
> + dev_set_drvdata(drm_dev->dev, dev);
> + drm_dev->dev_private = private;
> +
> + drm_mode_config_init(drm_dev);
> +
> + rockchip_drm_mode_config_init(drm_dev);
> +
> + dev->dma_parms = devm_kzalloc(dev, sizeof(*dev->dma_parms),
> + GFP_KERNEL);
> + if (!dev->dma_parms) {
> + ret = -ENOMEM;
> + goto err_config_cleanup;
> + }
> +
> + /* TODO(djkurtz): fetch the mapping start/size from somewhere */
> + mapping = arm_iommu_create_mapping(&platform_bus_type, 0x10000000,
> + SZ_1G);
> + if (IS_ERR(mapping)) {
> + ret = PTR_ERR(mapping);
> + goto err_config_cleanup;
> + }
> +
> + dma_set_coherent_mask(dev, DMA_BIT_MASK(32));
This is the default coherent mask. If you call this function, you
should normally check the return value, or call dma_set_mask first,
which you apparently don't do here, and in another place in this
patch.
Arnd
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox