eBPF sockhash datastructure and stream_parser/stream_verdict programs

eBPF sockhash datastructure and stream_parser/stream_verdict programs

[His Shadow]

Greeetings. Here's the problem. I've written a simple program, that,
when a connection is established, it establishes a connection to a
predetermined target and starts routing traffic between a user
connection and a new connection.
I've tried to use ebpf stream_parser/verdict programs for this,
however there's a problem: when a connection to my program is
established, client sends the data immediately, however there's a
delay, while I establish a connection to the target. So stream_verdict
never gets called, because the data is already in the socket receive
queue(or maybe I'm misunderstanding something). Is there a way around
this? Should I use something else, like skb_msg verdict?

Re: eBPF sockhash datastructure and stream_parser/stream_verdict programs

[Cong Wang]

Hi, His

On Mon, Jan 03, 2022 at 03:53:12PM +0300, His Shadow wrote:
> Greeetings. Here's the problem. I've written a simple program, that,
> when a connection is established, it establishes a connection to a
> predetermined target and starts routing traffic between a user
> connection and a new connection.
> I've tried to use ebpf stream_parser/verdict programs for this,
> however there's a problem: when a connection to my program is
> established, client sends the data immediately, however there's a
> delay, while I establish a connection to the target. So stream_verdict
> never gets called, because the data is already in the socket receive
> queue(or maybe I'm misunderstanding something). Is there a way around
> this? Should I use something else, like skb_msg verdict?

Are you saying the packets arrived before you put the socket
into the sockmap? If so, you can consider
BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB.

Hope this helps.

Thanks.

Fwd: eBPF sockhash datastructure and stream_parser/stream_verdict programs

[His Shadow]

Resending to the list, since gmail only picks first responder :(

>Are you saying the packets arrived before you put the socket into the sockmap?
Yes, exactly!

Could you elaborate on how BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB would
be helpful? I assume I need to set up a sockops program and record
passive ends pointers to bpf_sock somewhere, then redirect from
passive to passive or passive->active?


-- 
HisShadow

RE: Fwd: eBPF sockhash datastructure and stream_parser/stream_verdict programs

[John Fastabend]

His Shadow wrote:
> Resending to the list, since gmail only picks first responder :(
> 
> >Are you saying the packets arrived before you put the socket into the sockmap?
> Yes, exactly!
> 
> Could you elaborate on how BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB would
> be helpful? I assume I need to set up a sockops program and record
> passive ends pointers to bpf_sock somewhere, then redirect from
> passive to passive or passive->active?

Correct. The common way to build a bpf proxy here is to add sockets
to a sock{hash|map} from the sockops program when the connection
is established. This avoids missing bytes as you've noticed.

Alternatively, you can put the known sockets in the map from user
space and then monitor for new sockets with some tuple/key and
insert them based on whatever policy decides sockets need to
be redirected.

> 
> 
> -- 
> HisShadow

Re: Fwd: eBPF sockhash datastructure and stream_parser/stream_verdict programs

[His Shadow]

>Alternatively, you can put the known sockets in the map from user
>space and then monitor for new sockets with some tuple/key and
>insert them based on whatever policy decides sockets need to
>be redirected.

I think that's what I did. I put socket fds into a normal HASH, with
keys being localip localport remoteip remoteport for the other socket,
then in verdict I looked up the value for that socket's key, and
redirected based on that. But that's exactly when I encountered that
problem. Or are you talking about something else?

Re: Fwd: eBPF sockhash datastructure and stream_parser/stream_verdict programs

[His Shadow]

Let's say I'm writing a simple SOCKS4/4a server(yes, obsolete, but
still does it's job), no authorization(although it wouldn't matter).
So I need to read some data from the client: ip and port and maybe
domain if it's 4A. Then I connect to the ip/domain:port and send a
success response. Meanwhile the IP/Domain:port I connected to, before
sending success to the client might send some data, like if the target
was an SSH server, it sends a version string right away if you connect
to it. So it could be missed if I then put client and target socket
into a sockmap/hash and tell them to redirect traffic to each other.
I've tried very hard to figure out a way to use eBPF stream_verdict or
sk_msg_verdict to redirect traffic right when I need it, but I think
it is impossible. Do I have to do SOCKS4/4A parsing inside
stream_parser and communicate with my userspace program via a ring
buffer about the results? But I'm not sure how that could help with
missing bytes. I guess I got interested after I read that article on
cloudflare blog about sockmap
https://blog.cloudflare.com/sockmap-tcp-splicing-of-the-future/ , but
their example of an echo server is plagued with the same problem, if
you put a delay, before socket addition to a sockmap.