Segmentation fault on Downlink data transfer

Segmentation fault on Downlink data transfer

[Pargada, Carlos]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

Hi Marcel and Denis

There is an issue on downlink data transfer on ifx modem.

Basically, during a data transfer after PDP context activation (long data transfer 5MB), a segmentation fault  error could occurs at any time (even it is not systematic).
 GLib-CRITICAL **: g_io_channel_write_chars: assertion `channel != NULL' failed

Root cause of this issue is a ring buffer overflow (in gatio received data function). The overflow is detected and the DLC3 source context is destroyed in dispatch source function. After that the disconnect handler is called and try to destroyed the same context.

So on this issue there is 2 points:

1-      Management of the buffer overflow

2-      Buffer overflow cause

For the 1st point, I'm not sure that disconnect DLC channel is the solution in case of buffer overflow. I believe that is better to discard ongoing packet in the buffer. TCP could survived to packet loss and retransmit the packets. Throughput go down but the data pipe is not broken for the application.

For the 2nd point, ring buffer overflow happened because data is not consume in this buffer. This is a flow control problem. I suspect that for some reason the consumer is blocked (not schedule) during some time.
Ring buffer size is set to 4096 bytes. I realize some test with a ring buffer increase to 131072 bytes and I can get long file downlink transfer. Probably this size is too much high and  this size  buffer need to be tune.

Could you please give me feedback for this 2 points?

This issue is very similar to bug  track in: http://bugs.meego.com/show_bug.cgi?id=11700


Regards

Carlos










---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7346 bytes --]

Re: Segmentation fault on Downlink data transfer

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 2111 bytes --]

Hi Carlos,

> Root cause of this issue is a ring buffer overflow (in gatio received
> data function). The overflow is detected and the DLC3 source context is
> destroyed in dispatch source function. After that the disconnect handler
> is called and try to destroyed the same context.

In general a ring_buffer overflow indicates that the sender is sending
garbage or the consumer is doing something wrong.  Buffer overflows on
GAtIO should not happen.

> 
>  
> 
> So on this issue there is 2 points:
> 
> 1-      Management of the buffer overflow
> 
> 2-      Buffer overflow cause
> 
>                
> 
> For the 1^st point, I’m not sure that disconnect DLC channel is the
> solution in case of buffer overflow. I believe that is better to discard
> ongoing packet in the buffer. TCP could survived to packet loss and
> retransmit the packets. Throughput go down but the data pipe is not
> broken for the application.
> 
>  

Perhaps, but I think it is too early to speculate what we should do with
buffer overflows until we know the cause.

What is the exact setup that you're using?  Are you using GAtMux or the
kernel mux?

> 
> For the 2^nd point, ring buffer overflow happened because data is not
> consume in this buffer. This is a flow control problem. I suspect that
> for some reason the consumer is blocked (not schedule) during some time.
> 

The consumer is synchronous with GAtIO.  The receive_data function is
acting something like this:

- non-blocking reads until the buffer is full or nothing more to read
- Call the read handler
- Check for buffer overflow and kill the channel if it is.

> Ring buffer size is set to 4096 bytes. I realize some test with a ring
> buffer increase to 131072 bytes and I can get long file downlink
> transfer. Probably this size is too much high and  this size  buffer
> need to be tune.
> 

So to me this is sounding like a problem in GAtRawIP implementation.
Can you check whether using synchronous writes to the TUN device fix
your problem?  E.g. in new_bytes of gatrawip.c.

Regards,
-Denis

RE: Segmentation fault on Downlink data transfer

[Bastian, Waldo]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

> Hi Marcel and Denis
>
> There is an issue on downlink data transfer on ifx modem.

Please note that ifx modem will switch over to in-kernel MUX for data handling once
the kernel MUX driver enhancements for network interface support have been accepted
into MeeGo.

Cheers,
Waldo

Re: Segmentation fault on Downlink data transfer

[Pargada, Carlos]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

Hi Denis,

> What is the exact setup that you're using?  Are you using GAtMux or the kernel mux?

Actually the mux was the internal to oFono, Kernel MUX is planned for further versions.


> Can you check whether using synchronous writes to the TUN device fix your problem?  E.g. in new_bytes of gatrawip.c.

Changing priority from G_PRIORITY_NORMAL to G_PRIORITY_HIGH avoid the issue of ring buffer overflow.
However  the ring buffer usage is at 2/3. This mean that 2 packets of 1420 are stored. A 3 one crash the system.

In Mobile data transfer, it is often the cases that  latency of packets can be much more greater that in wired network (RLC layer in Modem buffers incoming packet to reassemble its and deliver packet in right order of reception). In this case,  several packet can be delivered by the Modem in burst. If another packet are to closed, we get a new ring buffer overflow. This is why  I believe that this buffer need to be increase to avoid this situation.
Taking some logs (tcpdump, ofono and internal Modem logs) show that burst of packets often occurs.

So to Fix this issue 2 actions is required:


-          Set the G_PRIORITY_HIGH instead of NORMAL  in g_io_add_watch_full  in g_io_set_write_handler function (avoid to change mechanism to synchronous mode)

-          Secure the ring buffer overflow increasing the size (for example to 7100: capacity to store 5 Packets of 1420 bytes seems enough).

> For the dlc disconnection after a ring buffer overflow, I continue to suggest to change the behavior. At the end user side, a shutdown of the Modem is not the solution. There is not a Modem issue and  all others functionalities of the Modem not need to be deactivate (ongoing Call drop if ongoing, No  Emergency Call available during reinitialisation, etc ...)

Give me your feedback for the fix and tell me If you want me to submit a patch with this proposition.

Regards

Carlos










---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 10474 bytes --]