conntrack EILSEQ followed by ENOBUFS

conntrack EILSEQ followed by ENOBUFS

[abirvalg]

On Linux wwwwww-701SD 2.6.35-30-generic #59-Ubuntu SMP Tue Aug 30
15:58:00 UTC 2011 i686 GNU/Linux
with libnetfilter-conntrack3 Version: 0.0.101-1 

Hi, my application uses libnetfiler_conntrack. When I want to set a mark
on a connection, I 
nfct_set_attr_* all the necessary fields of struct nf_conntrack* and
then do
nfct_query(setmark_handle, NFCT_Q_GET, ct)

setmark_handle has a callback registered thusly:
   if ((nfct_callback_register(setmark_handle, NFCT_T_ALL, setmark,
NULL) == -1)) {perror("cb_reg");}

So ct object lands here:

int setmark (enum nf_conntrack_msg_type type, struct nf_conntrack
*mct,void *data){
  nfct_set_attr_u32(mct, ATTR_MARK, nfmark_to_set);
  nfct_query(setmark_handle, NFCT_Q_UPDATE, mct);   ***
  return NFCT_CB_CONTINUE;
}

All works fine and dandy and i can see with "conntrack -L" marks being
set. At times I get EBUSY from nfct_query(...NFCT_Q_GET...) but I simply call nfct_query(...NFCT_Q_GET...) again and the query goes through.
Until at a seemingly random point, I start getting:
EILSEQ from nfct_query(...NFCT_Q_GET...) (Invalid or incomplete multibyte  or wide
character). I don't resend this packet with nfct_query(...NFCT_Q_GET...), From then on every single nfct_query(...NFCT_Q_GET...) returns EILSEQ maybe for a 100 or so queries, until I finally get ENOBUFS and my app hangs.
Even calling "conntrack -L" at that point hangs - no output displayed
and prog doesn't return.

I hope that the line in the code above with *** is not the offending
one:NFCT_Q_UPDATE doesn't technically require a handle, yet the API says
it should be there, so I put the handle of this very callback.

Please let me know if there is any more info I could provide you with. I
am also willing to install conntrack_dbg package and investigate the
issue if need be.
Thanks.

Re: conntrack EILSEQ followed by ENOBUFS

[Pablo Neira Ayuso]

On Mon, Oct 10, 2011 at 09:17:02PM +0000, abirvalg@lavabit.com wrote:
> On Linux wwwwww-701SD 2.6.35-30-generic #59-Ubuntu SMP Tue Aug 30
> 15:58:00 UTC 2011 i686 GNU/Linux
> with libnetfilter-conntrack3 Version: 0.0.101-1 
> 
> Hi, my application uses libnetfiler_conntrack. When I want to set a mark
> on a connection, I 
> nfct_set_attr_* all the necessary fields of struct nf_conntrack* and
> then do
> nfct_query(setmark_handle, NFCT_Q_GET, ct)

If the connection already exists,you have to use NFCT_Q_UPDATE.

> setmark_handle has a callback registered thusly:
>    if ((nfct_callback_register(setmark_handle, NFCT_T_ALL, setmark,
> NULL) == -1)) {perror("cb_reg");}
> 
> So ct object lands here:
> 
> int setmark (enum nf_conntrack_msg_type type, struct nf_conntrack
> *mct,void *data){
>   nfct_set_attr_u32(mct, ATTR_MARK, nfmark_to_set);
>   nfct_query(setmark_handle, NFCT_Q_UPDATE, mct);   ***
>   return NFCT_CB_CONTINUE;
>
> All works fine and dandy and i can see with "conntrack -L" marks being
> set. At times I get EBUSY from nfct_query(...NFCT_Q_GET...) but I
> simply call nfct_query(...NFCT_Q_GET...) again and the query goes
> through.

EBUSY shouldn't happen unless you are playing with the conntrack
flags or trying to assign some conntrack helper.

In that case, I'd need some example code that can trigger this error.

> Until at a seemingly random point, I start getting:
> EILSEQ from nfct_query(...NFCT_Q_GET...) (Invalid or incomplete multibyte  or wide
> character). I don't resend this packet with nfct_query(...NFCT_Q_GET...), From then on every single nfct_query(...NFCT_Q_GET...) returns EILSEQ maybe for a 100 or so queries, until I finally get ENOBUFS and my app hangs.
> Even calling "conntrack -L" at that point hangs - no output displayed
> and prog doesn't return.

> I hope that the line in the code above with *** is not the offending
> one:NFCT_Q_UPDATE doesn't technically require a handle, yet the API says
> it should be there, so I put the handle of this very callback.
> 
> Please let me know if there is any more info I could provide you with. I
> am also willing to install conntrack_dbg package and investigate the
> issue if need be.

Regarding the EILSEQ error:

The second parameter of nfct_open must be 0. However, if you use the
same socket for sending commands and receiving events, then you have
to disable sequence tracking, there is a function in libnfnetlink to
do that.

Re: conntrack EILSEQ followed by ENOBUFS

[Pablo Neira Ayuso]

On Thu, Oct 13, 2011 at 12:50:55PM +0000, abirvalg@lavabit.com wrote:
> Sorry for not being sufficiently specific. Since you showed more interest, I feel empowered to go into further detail.

You're reporting problems, I'm not a wizard, if you don't provide
details, it's way harder for me to guess what's wrong ;-)

> My app uses libnetfilter_queue. When it NF_ACCEPTs a packet, it immediately goes on to set a mark on the connection of which the ACCEPTed packet is part of. But in order to set the mark on the connection, it first NFCT_Q_GETs that conntrack (because it knows srcIP, destIP, srcPort, destPort, L4proto, L3proto). Once it got the conntrack, it can proceed to set a mark on it using NFCT_Q_UPDATE.
> 
> I'm hitting EBUSY with NFCT_Q_GET.
> My impression was that it was ok to hit EBUSY, since I'm making such a heavy use of conntrack table, contantly updating in. Besides I have watch -n 1 'conntrack -L' running in another console. So it's double pressure.
> It may be beside the point, but apart from this GETing and UPDATEing, I have another thread (that uses a different handle) which every minute or so does NFCT_Q_DUMP (that's 1000+ entries) and the handle's callback then NFCT_Q_DESTROYS (based on the nfmark) approx. 80% of the dump. 

There are different situations in which you may hit EBUSY:

For example, if you call NFCT_Q_DUMP after some unconcluded dumping
(one previous NFCT_Q_DUMP operation that is still on the way).

Regarding threads, it's a bad idea to share conntrack handlers between
different threads.

As said, I need some code to look at for better helping.

Re: conntrack EILSEQ followed by ENOBUFS

[abirvalg]

Gracias for responding, Pablo.
My problem has now scaled down by 50%. EILSEQ happened due to a race when 2 threads in my app set_attr* to the same stuct nf_conntrack simultaneously.
I only have EBUSY error occasionally. I now upgraded to libnetfilter_conntrack 0.9.1 and the frequency of EBUSY has dropped significantly. I seed a torrent which creates 30 NEW connections per second and leave the machine running for 24  hours.
I put a mark on each of those NEW connection.  I only got 1 EBUSY so far.

Please let me know if you are still interested in getting to the bottom of that 1 EBUSY per 24 hours.

>EBUSY shouldn't happen unless you are playing with the conntrack
>flags or trying to assign some conntrack helper.
>In that case, I'd need some example code that can trigger this error.

No, I don't have any conntrack helpers. And I'm not touching any conntrack flags. Just doing nfct_query(...NFCT_Q_GET...).

Regards.




On Wed, 12 Oct 2011 18:16:15 +0200
Pablo Neira Ayuso <pablo@netfilter.org> wrote:

> On Mon, Oct 10, 2011 at 09:17:02PM +0000, abirvalg@lavabit.com wrote:
> > On Linux wwwwww-701SD 2.6.35-30-generic #59-Ubuntu SMP Tue Aug 30
> > 15:58:00 UTC 2011 i686 GNU/Linux
> > with libnetfilter-conntrack3 Version: 0.0.101-1 
> > 
> > Hi, my application uses libnetfiler_conntrack. When I want to set a mark
> > on a connection, I 
> > nfct_set_attr_* all the necessary fields of struct nf_conntrack* and
> > then do
> > nfct_query(setmark_handle, NFCT_Q_GET, ct)
> 
> If the connection already exists,you have to use NFCT_Q_UPDATE.
> 
> > setmark_handle has a callback registered thusly:
> >    if ((nfct_callback_register(setmark_handle, NFCT_T_ALL, setmark,
> > NULL) == -1)) {perror("cb_reg");}
> > 
> > So ct object lands here:
> > 
> > int setmark (enum nf_conntrack_msg_type type, struct nf_conntrack
> > *mct,void *data){
> >   nfct_set_attr_u32(mct, ATTR_MARK, nfmark_to_set);
> >   nfct_query(setmark_handle, NFCT_Q_UPDATE, mct);   ***
> >   return NFCT_CB_CONTINUE;
> >
> > All works fine and dandy and i can see with "conntrack -L" marks being
> > set. At times I get EBUSY from nfct_query(...NFCT_Q_GET...) but I
> > simply call nfct_query(...NFCT_Q_GET...) again and the query goes
> > through.
> 
> EBUSY shouldn't happen unless you are playing with the conntrack
> flags or trying to assign some conntrack helper.
> 
> In that case, I'd need some example code that can trigger this error.
> 
> > Until at a seemingly random point, I start getting:
> > EILSEQ from nfct_query(...NFCT_Q_GET...) (Invalid or incomplete multibyte  or wide
> > character). I don't resend this packet with nfct_query(...NFCT_Q_GET...), From then on every single nfct_query(...NFCT_Q_GET...) returns EILSEQ maybe for a 100 or so queries, until I finally get ENOBUFS and my app hangs.
> > Even calling "conntrack -L" at that point hangs - no output displayed
> > and prog doesn't return.
> 
> > I hope that the line in the code above with *** is not the offending
> > one:NFCT_Q_UPDATE doesn't technically require a handle, yet the API says
> > it should be there, so I put the handle of this very callback.
> > 
> > Please let me know if there is any more info I could provide you with. I
> > am also willing to install conntrack_dbg package and investigate the
> > issue if need be.
> 
> Regarding the EILSEQ error:
> 
> The second parameter of nfct_open must be 0. However, if you use the
> same socket for sending commands and receiving events, then you have
> to disable sequence tracking, there is a function in libnfnetlink to
> do that.

Re: conntrack EILSEQ followed by ENOBUFS

[abirvalg]

Sorry for not being sufficiently specific. Since you showed more interest, I feel empowered to go into further detail.

My app uses libnetfilter_queue. When it NF_ACCEPTs a packet, it immediately goes on to set a mark on the connection of which the ACCEPTed packet is part of. But in order to set the mark on the connection, it first NFCT_Q_GETs that conntrack (because it knows srcIP, destIP, srcPort, destPort, L4proto, L3proto). Once it got the conntrack, it can proceed to set a mark on it using NFCT_Q_UPDATE.

I'm hitting EBUSY with NFCT_Q_GET.
My impression was that it was ok to hit EBUSY, since I'm making such a heavy use of conntrack table, contantly updating in. Besides I have watch -n 1 'conntrack -L' running in another console. So it's double pressure.

It may be beside the point, but apart from this GETing and UPDATEing, I have another thread (that uses a different handle) which every minute or so does NFCT_Q_DUMP (that's 1000+ entries) and the handle's callback then NFCT_Q_DESTROYS (based on the nfmark) approx. 80% of the dump. 




On Thu, 13 Oct 2011 11:30:14 +0200
Pablo Neira Ayuso <pablo@netfilter.org> wrote:

> On Thu, Oct 13, 2011 at 11:10:20AM +0000, abirvalg@lavabit.com wrote:
> > Gracias for responding, Pablo.
> > My problem has now scaled down by 50%. EILSEQ happened due to a race when 2 threads in my app set_attr* to the same stuct nf_conntrack simultaneously.
> 
> Hm, you didn't mention you were using threads. Then, it's normal to
> run into sequence tracking issues if both are using the same socket.
> It can be a good idea to give a try to use two different sockets, one
> per thread.
> 
> > I only have EBUSY error occasionally. I now upgraded to libnetfilter_conntrack 0.9.1 and the frequency of EBUSY has dropped significantly. I seed a torrent which creates 30 NEW connections per second and leave the machine running for 24  hours.
> > I put a mark on each of those NEW connection.  I only got 1 EBUSY so far.
> > 
> > Please let me know if you are still interested in getting to the bottom of that 1 EBUSY per 24 hours.
> 
> I'd need to know more information on what you're doing. Right now, I
> don't understand what you're doing further than "updating one ct mark".
> 
> Again, some example code that I can look at, or some description would
> help me a lot.
> 
> > >EBUSY shouldn't happen unless you are playing with the conntrack
> > >flags or trying to assign some conntrack helper.
> > >In that case, I'd need some example code that can trigger this error.
> > 
> > No, I don't have any conntrack helpers. And I'm not touching any conntrack flags. Just doing nfct_query(...NFCT_Q_GET...).
> 
> You're hitting EBUSY with NFCT_Q_GET or NFCT_Q_UPDATE?

Re: conntrack EILSEQ followed by ENOBUFS

[abirvalg]

On Thu, 13 Oct 2011 13:02:10 +0200
Pablo Neira Ayuso <pablo@netfilter.org> wrote:

> On Thu, Oct 13, 2011 at 12:50:55PM +0000, abirvalg@lavabit.com wrote:
> > Sorry for not being sufficiently specific. Since you showed more interest, I feel empowered to go into further detail.
> 
> You're reporting problems, I'm not a wizard, if you don't provide
> details, it's way harder for me to guess what's wrong ;-)
> 
> > My app uses libnetfilter_queue. When it NF_ACCEPTs a packet, it immediately goes on to set a mark on the connection of which the ACCEPTed packet is part of. But in order to set the mark on the connection, it first NFCT_Q_GETs that conntrack (because it knows srcIP, destIP, srcPort, destPort, L4proto, L3proto). Once it got the conntrack, it can proceed to set a mark on it using NFCT_Q_UPDATE.
> > 
> > I'm hitting EBUSY with NFCT_Q_GET.
> > My impression was that it was ok to hit EBUSY, since I'm making such a heavy use of conntrack table, contantly updating in. Besides I have watch -n 1 'conntrack -L' running in another console. So it's double pressure.
> > It may be beside the point, but apart from this GETing and UPDATEing, I have another thread (that uses a different handle) which every minute or so does NFCT_Q_DUMP (that's 1000+ entries) and the handle's callback then NFCT_Q_DESTROYS (based on the nfmark) approx. 80% of the dump. 
> 
> There are different situations in which you may hit EBUSY:
> 
> For example, if you call NFCT_Q_DUMP after some unconcluded dumping
> (one previous NFCT_Q_DUMP operation that is still on the way).
> 
> Regarding threads, it's a bad idea to share conntrack handlers between
> different threads.
> 
> As said, I need some code to look at for better helping.

Sorry for not being forthcoming with the code, I forgot that my project has gitweb enabled,
Here is the offending line: (line 1541)

http://leopardflower.git.sourceforge.net/git/gitweb.cgi?p=leopardflower/leopardflower;a=blob;f=lpfw.c;h=a95e80359b5b3c00657d1e695ff273e84e8288d1;hb=8e787ce7e0800c1541b9427e517d14d8ae55e756#l1541

Cheers.