Fw: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?

Fw: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?

[Andrew Morton]

Begin forwarded message:

Date: Mon, 31 May 2004 14:45:08 -0400
From: Russell Leighton <russ@elegant-software.com>
To: linux-kernel@vger.kernel.org
Subject: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?



I have a program that works fine under stock rh9 (2.4.2-8) but has 
issues getting signaled under FedoraCore2 (2.6.5-1.358)
using SETSIG to a Posix RT signal.

The program does the standard:

  /* hook to process */
  if ( fcntl(fdcallback->fd, F_SETOWN, mon->handler_q.thread->pid) == -1 ) {
    aw_log(fdcallback->handler->logger, AW_ERROR_LOG_LEVEL,
       "cannot set owner on fd (%s)",
       strerror(errno));
  }/* end if */

  /* make async */
  if ( fcntl(fdcallback->fd, F_SETFL, (O_NONBLOCK | O_ASYNC) ) == -1 ) {
    aw_log(fdcallback->handler->logger, AW_ERROR_LOG_LEVEL,
       "cannot set async on fd (%s)",
       strerror(errno));
  }/* end if */

  /* hook to signal */
  if ( fcntl(fdcallback->fd, F_SETSIG, AW_SIG_FD) == -1 ) {
    aw_log(fdcallback->handler->logger, AW_ERROR_LOG_LEVEL,
       "cannot set signal on fd (%s)",
       strerror(errno));
  }/* end if */

Under Fedora things work well for raw sockets (much lower latency than 
in 2.4!) but are inconsistent with udp or tcp sockets.

In the udp case, I when I listen for multicast packets my app only 
receives them when I am running a tcpdump (bizarre!).

In the tcp case, I don't get signaled if I do the F_SETSIG on more than 
1 fd.

Any tips on tracking this down would be much appreciated.

Thx

Russ

Re: Fw: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?

[David Stevens]

> In the udp case, I when I listen for multicast packets my app only
> receives them when I am running a tcpdump (bizarre!).

Russ,
        This piece (which I expect has nothing to do with the other
problems you mentioned) sounds like you haven't joined the groups on
the interface on which you're receiving the multicast packets.
"tcpdump" will place the interface in "promiscuous mode" which will
receive all packets, and ordinary packet delivery will allow
the application to receive them, even if you haven't joined the group
on the relevant interface.
        To verify if the group joins is broken, you can look at
/proc/net/igmp. If the groups you're joining are not listed on the
interface you want, the program isn't joining the groups correctly.
Group membership is per-interface, so joining a group on one interface
does not join it on another.
        Feel free to contact me if you need some help debugging the
multicast problem.

                                                        +-DLS

Re: Fw: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?

[Russell Leighton]

[-- Attachment #1: Type: text/plain, Size: 4110 bytes --]

Thanks for the suggestion.

I tried using the interface itself and INADDR_ANY...the signals are not 
being received under 2.6 (FedoraCore2) UNLESS tcpdump is running...note 
this works fine under 2.4.

Below is the code fragment that sets up the socket (the previous email 
had the code fragment that sets up the posix rt signals on the fd), any 
help would be greatly appreciated:

  /* mc_fd */
  {

    /* make it */
    if ( (h->state.mc_fd = socket(AF_INET, SOCK_DGRAM, 0)) == -1 ) {
      aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL, "mcrxhandler 
cannot create socket: %s", strerror(errno));
      goto error;
    }

    /* set it to nonblocking so that this can be async */
    if ( fcntl(h->state.mc_fd,  F_SETFL, O_NONBLOCK)  == -1) {
      aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL, "mcrxhandler 
cannot set socket nonblocking: %s", strerror(errno));
      goto error;
    }

    /* set the mc interface */
    if ( setsockopt(h->state.mc_fd, IPPROTO_IP, IP_MULTICAST_IF, 
&(h->mcast_if_addr), sizeof(h->mcast_if_addr)) < 0 ) {
      u_int8_t
    *ip = (u_int8_t *)&(h->mcast_if_addr);

      aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL,
         "mcrxhandler cannot set socket IP_MULTICAST_IF for %u.%u.%u.%u: 
%s",
         ip[0],ip[1],ip[2],ip[4],
         strerror(errno));
      goto error;
    }

    { /* debugging where packets are going when you have many interfaces 
is a pain, you want to log this! */
      u_int8_t
    *ip = (u_int8_t *)&(h->mcast_if_addr);
      aw_log(h->handler_header.logger,  AW_INFO_LOG_LEVEL, "mcrxhandler 
running multicast on interface %u.%u.%u.%u",
         ip[0],ip[1],ip[2],ip[3]);
    }

    /* use setsockopt() to request that the kernel join a multicast group */
    {
      struct ip_mreq
    mreq;

      /* set up */
      memset(&mreq, 0 , sizeof(mreq));
      mreq.imr_multiaddr.s_addr= h->mcast_grp_addr.sin_addr.s_addr;
      mreq.imr_interface.s_addr= h->mcast_if_addr.s_addr;

      if (setsockopt(h->state.mc_fd, IPPROTO_IP,IP_ADD_MEMBERSHIP, (char 
*)&mreq, sizeof(mreq)) < 0) {
    aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL,
           "mcrxhandler cannot set socket IPPROTO_IP,IP_ADD_MEMBERSHIP: 
%s", strerror(errno));
    goto error;
      }
    }

    /* so we can have many processes listening to mcast */
    {
      int32_t
    itmp = 1;
      if ( setsockopt(h->state.mc_fd, SOL_SOCKET, SO_REUSEADDR , (char 
*)&itmp, sizeof(itmp)) < 0 ) {
    aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL, "mcrxhandler 
cannot set socket SO_REUSEADDR: %s", strerror(errno));
      }
    }

    /* bind to receive messages */
    if (bind(h->state.mc_fd, (struct sockaddr *)&(h->mcast_grp_addr), 
sizeof(h->mcast_grp_addr)) < 0) {
      aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL, "mcrxhandler 
cannot bind to multicast socket", strerror(errno));
      goto error ;
    }

    /* add callback to handle packets on mc_fd */
    aw_add_handler_fdcallback(mon, (aw_handler_t *)h, h->state.mc_fd, 
do_read);

  } /* end mc_fd */




David Stevens wrote:

>>In the udp case, I when I listen for multicast packets my app only
>>receives them when I am running a tcpdump (bizarre!).
>>    
>>
>
>Russ,
>        This piece (which I expect has nothing to do with the other
>problems you mentioned) sounds like you haven't joined the groups on
>the interface on which you're receiving the multicast packets.
>"tcpdump" will place the interface in "promiscuous mode" which will
>receive all packets, and ordinary packet delivery will allow
>the application to receive them, even if you haven't joined the group
>on the relevant interface.
>        To verify if the group joins is broken, you can look at
>/proc/net/igmp. If the groups you're joining are not listed on the
>interface you want, the program isn't joining the groups correctly.
>Group membership is per-interface, so joining a group on one interface
>does not join it on another.
>        Feel free to contact me if you need some help debugging the
>multicast problem.
>
>                                                        +-DLS
>
>
>
>  
>


[-- Attachment #2: Type: text/html, Size: 6117 bytes --]

Re: Fw: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?

[Russell Leighton]

I forgot to answer your question, I confirmed via the proc interface 
that the group has been joined.

I am thinking the issue is related to F_SETSIG. I don't read() until I 
get a signal and I am not getting ANY signals for the multicast data.

Googling around a little I saw changes in the futex code around FUTEX_FD 
... perhaps there is a bug?

Cobbling together a small test piece of code is the next thing to do...


Russell Leighton wrote:

> Thanks for the suggestion.
>
> I tried using the interface itself and INADDR_ANY...the signals are 
> not being received under 2.6 (FedoraCore2) UNLESS tcpdump is 
> running...note this works fine under 2.4.
>
> Below is the code fragment that sets up the socket (the previous email 
> had the code fragment that sets up the posix rt signals on the fd), 
> any help would be greatly appreciated:
>
>   /* mc_fd */
>   {
>
>     /* make it */
>     if ( (h->state.mc_fd = socket(AF_INET, SOCK_DGRAM, 0)) == -1 ) {
>       aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL, 
> "mcrxhandler cannot create socket: %s", strerror(errno));
>       goto error;
>     }
>
>     /* set it to nonblocking so that this can be async */
>     if ( fcntl(h->state.mc_fd,  F_SETFL, O_NONBLOCK)  == -1) {
>       aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL, 
> "mcrxhandler cannot set socket nonblocking: %s", strerror(errno));
>       goto error;
>     }
>
>     /* set the mc interface */
>     if ( setsockopt(h->state.mc_fd, IPPROTO_IP, IP_MULTICAST_IF, 
> &(h->mcast_if_addr), sizeof(h->mcast_if_addr)) < 0 ) {
>       u_int8_t
>     *ip = (u_int8_t *)&(h->mcast_if_addr);
>
>       aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL,
>          "mcrxhandler cannot set socket IP_MULTICAST_IF for 
> %u.%u.%u.%u: %s",
>          ip[0],ip[1],ip[2],ip[4],
>          strerror(errno));
>       goto error;
>     }
>
>     { /* debugging where packets are going when you have many 
> interfaces is a pain, you want to log this! */
>       u_int8_t
>     *ip = (u_int8_t *)&(h->mcast_if_addr);
>       aw_log(h->handler_header.logger,  AW_INFO_LOG_LEVEL, 
> "mcrxhandler running multicast on interface %u.%u.%u.%u",
>          ip[0],ip[1],ip[2],ip[3]);
>     }
>
>     /* use setsockopt() to request that the kernel join a multicast 
> group */
>     {
>       struct ip_mreq
>     mreq;
>
>       /* set up */
>       memset(&mreq, 0 , sizeof(mreq));
>       mreq.imr_multiaddr.s_addr= h->mcast_grp_addr.sin_addr.s_addr;
>       mreq.imr_interface.s_addr= h->mcast_if_addr.s_addr;
>
>       if (setsockopt(h->state.mc_fd, IPPROTO_IP,IP_ADD_MEMBERSHIP, 
> (char *)&mreq, sizeof(mreq)) < 0) {
>     aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL,
>            "mcrxhandler cannot set socket 
> IPPROTO_IP,IP_ADD_MEMBERSHIP: %s", strerror(errno));
>     goto error;
>       }
>     }
>
>     /* so we can have many processes listening to mcast */
>     {
>       int32_t
>     itmp = 1;
>       if ( setsockopt(h->state.mc_fd, SOL_SOCKET, SO_REUSEADDR , (char 
> *)&itmp, sizeof(itmp)) < 0 ) {
>     aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL, "mcrxhandler 
> cannot set socket SO_REUSEADDR: %s", strerror(errno));
>       }
>     }
>
>     /* bind to receive messages */
>     if (bind(h->state.mc_fd, (struct sockaddr *)&(h->mcast_grp_addr), 
> sizeof(h->mcast_grp_addr)) < 0) {
>       aw_log(h->handler_header.logger, AW_ERROR_LOG_LEVEL, 
> "mcrxhandler cannot bind to multicast socket", strerror(errno));
>       goto error ;
>     }
>
>     /* add callback to handle packets on mc_fd */
>     aw_add_handler_fdcallback(mon, (aw_handler_t *)h, h->state.mc_fd, 
> do_read);
>
>   } /* end mc_fd */
>
>
>
>
> David Stevens wrote:
>
>>>In the udp case, I when I listen for multicast packets my app only
>>>receives them when I am running a tcpdump (bizarre!).
>>>    
>>>
>>
>>Russ,
>>        This piece (which I expect has nothing to do with the other
>>problems you mentioned) sounds like you haven't joined the groups on
>>the interface on which you're receiving the multicast packets.
>>"tcpdump" will place the interface in "promiscuous mode" which will
>>receive all packets, and ordinary packet delivery will allow
>>the application to receive them, even if you haven't joined the group
>>on the relevant interface.
>>        To verify if the group joins is broken, you can look at
>>/proc/net/igmp. If the groups you're joining are not listed on the
>>interface you want, the program isn't joining the groups correctly.
>>Group membership is per-interface, so joining a group on one interface
>>does not join it on another.
>>        Feel free to contact me if you need some help debugging the
>>multicast problem.
>>
>>                                                        +-DLS
>>
>>
>>
>>  
>>
>

Re: Fw: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?

[Russell Leighton]

Thanks to all that helped me troubleshoot.

Of the 2 issues I had with FedoraCore2, one problem is solved:

    * Multicast issues were solved by using another NIC. It seems that
      the driver for the NatSemi DP8381[56] does not receive mutlicast
      properly.
    * F_SETSIG still seems broken for TCP for me when my process sets up
      more than a few fd's...I will try the latest kernel to see if this
      goes away


Russ

Andrew Morton wrote:

>Begin forwarded message:
>
>Date: Mon, 31 May 2004 14:45:08 -0400
>From: Russell Leighton <russ@elegant-software.com>
>To: linux-kernel@vger.kernel.org
>Subject: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?
>
>
>
>I have a program that works fine under stock rh9 (2.4.2-8) but has 
>issues getting signaled under FedoraCore2 (2.6.5-1.358)
>using SETSIG to a Posix RT signal.
>
>The program does the standard:
>
>  /* hook to process */
>  if ( fcntl(fdcallback->fd, F_SETOWN, mon->handler_q.thread->pid) == -1 ) {
>    aw_log(fdcallback->handler->logger, AW_ERROR_LOG_LEVEL,
>       "cannot set owner on fd (%s)",
>       strerror(errno));
>  }/* end if */
>
>  /* make async */
>  if ( fcntl(fdcallback->fd, F_SETFL, (O_NONBLOCK | O_ASYNC) ) == -1 ) {
>    aw_log(fdcallback->handler->logger, AW_ERROR_LOG_LEVEL,
>       "cannot set async on fd (%s)",
>       strerror(errno));
>  }/* end if */
>
>  /* hook to signal */
>  if ( fcntl(fdcallback->fd, F_SETSIG, AW_SIG_FD) == -1 ) {
>    aw_log(fdcallback->handler->logger, AW_ERROR_LOG_LEVEL,
>       "cannot set signal on fd (%s)",
>       strerror(errno));
>  }/* end if */
>
>Under Fedora things work well for raw sockets (much lower latency than 
>in 2.4!) but are inconsistent with udp or tcp sockets.
>
>In the udp case, I when I listen for multicast packets my app only 
>receives them when I am running a tcpdump (bizarre!).
>
>In the tcp case, I don't get signaled if I do the F_SETSIG on more than 
>1 fd.
>
>Any tips on tracking this down would be much appreciated.
>
>Thx
>
>Russ
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>

Re: Fw: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?

[Russell Leighton]

Last issue resolved. I am posting so there is a complete thread for 
anyone that stumbles on this in the future.

F_SETSIG IS working...what is NOT working (as it did in 2.4) is the 
interaction between clone() and getpid() in child processes.
This issue altered the control of my program and made it look like I was 
not receiving signals.

I'll send another email to the kernel mailing list w/a test program that 
demonstrates this.

Russell Leighton wrote:

>
> Thanks to all that helped me troubleshoot.
>
> Of the 2 issues I had with FedoraCore2, one problem is solved:
>
>    * Multicast issues were solved by using another NIC. It seems that
>      the driver for the NatSemi DP8381[56] does not receive mutlicast
>      properly.
>    * F_SETSIG still seems broken for TCP for me when my process sets up
>      more than a few fd's...I will try the latest kernel to see if this
>      goes away
>
>
> Russ
>
> Andrew Morton wrote:
>
>> Begin forwarded message:
>>
>> Date: Mon, 31 May 2004 14:45:08 -0400
>> From: Russell Leighton <russ@elegant-software.com>
>> To: linux-kernel@vger.kernel.org
>> Subject: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?
>>
>>
>>
>> I have a program that works fine under stock rh9 (2.4.2-8) but has 
>> issues getting signaled under FedoraCore2 (2.6.5-1.358)
>> using SETSIG to a Posix RT signal.
>>
>> The program does the standard:
>>
>>  /* hook to process */
>>  if ( fcntl(fdcallback->fd, F_SETOWN, mon->handler_q.thread->pid) == 
>> -1 ) {
>>    aw_log(fdcallback->handler->logger, AW_ERROR_LOG_LEVEL,
>>       "cannot set owner on fd (%s)",
>>       strerror(errno));
>>  }/* end if */
>>
>>  /* make async */
>>  if ( fcntl(fdcallback->fd, F_SETFL, (O_NONBLOCK | O_ASYNC) ) == -1 ) {
>>    aw_log(fdcallback->handler->logger, AW_ERROR_LOG_LEVEL,
>>       "cannot set async on fd (%s)",
>>       strerror(errno));
>>  }/* end if */
>>
>>  /* hook to signal */
>>  if ( fcntl(fdcallback->fd, F_SETSIG, AW_SIG_FD) == -1 ) {
>>    aw_log(fdcallback->handler->logger, AW_ERROR_LOG_LEVEL,
>>       "cannot set signal on fd (%s)",
>>       strerror(errno));
>>  }/* end if */
>>
>> Under Fedora things work well for raw sockets (much lower latency 
>> than in 2.4!) but are inconsistent with udp or tcp sockets.
>>
>> In the udp case, I when I listen for multicast packets my app only 
>> receives them when I am running a tcpdump (bizarre!).
>>
>> In the tcp case, I don't get signaled if I do the F_SETSIG on more 
>> than 1 fd.
>>
>> Any tips on tracking this down would be much appreciated.
>>
>> Thx
>>
>> Russ
>>
>> -
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>>
>>
>>  
>>
>
>

Re: Fw: F_SETSIG broken/changed in 2.6 for UDP and TCP sockets?

[Manfred Spraul]

Russell Leighton wrote:

>> Thanks to all that helped me troubleshoot.
>>
>> Of the 2 issues I had with FedoraCore2, one problem is solved:
>>
>>    * Multicast issues were solved by using another NIC. It seems that
>>      the driver for the NatSemi DP8381[56] does not receive mutlicast
>>      properly.
>  
>

Odd. I've just tried the driver from 2.6.7-rc2 and multicast receive 
works: the hardware correctly accepts packets for the requested group 
and skips packets for all other groups.
Which app do you use for multicast receive/send? Which nic do you use 
now instead of natsemi?

--
    Manfred