Call netfilter okfn on stolen packets

Call netfilter okfn on stolen packets

[Federico De Marchi]

Hello,

I am trying to write a kernel module using netfilter to be able to stall 
and store TCP packets on data structures of my own and then release them 
on a signal. At first I used the nf_queue+nf_reinject system, but now I 
would rather be able to manage packets and data structures more freely 
in kernel space.

I am using kernel version 5.10.113 and using only a NF_INET_POST_ROUTING 
hook set to INT_MAX priority. Trying to copy what nf_queue/nf_reinject 
do, this is what I do for storing, before returning NF_STOLEN in the 
hook function:

     pkt.skb = skb;
     pkt.state = *state;
     rcu_read_lock();
     if (state->in)
         dev_hold(state->in);
     if (state->out)
         dev_hold(state->out);
     if (state->sk)
         sock_hold(state->sk);
     rcu_read_unlock();

And this is what I do for releasing:

     struct nf_hook_state *state = &pkt.state;
     local_bh_disable();
     (state->okfn)(state->net, state->sk, pkt.skb);
     local_bh_enable();
     rcu_read_lock();
     if (state->in)
         dev_put(state->in);
     if (state->out)
         dev_put(state->out);
     if (state->sk)
         sock_put(state->sk);
     rcu_read_unlock();

And this stalling/releasing works well until at some point it does not 
and my VM freezes after calling the okfn (which in my case always is 
ip_finish_output if I understood correctly). I am debugging by 
outputting dmesg prints from guest to host and the kernel throws no 
error, it is just frozen. I'm also not getting any feedback from KASAN. 
So I guess I might be doing something wrong resulting in a deadlock of 
some sort? Does anybody have any pointer whether there is anything else 
I should be looking out for when stalling and then releasing a packet 
this way?

Thank you for your time,

Federico