Sleep and Wake up

Sleep and Wake up

[Abu Rasheda]

I am testing my driver on much faster host processor and facing
following issues:

My host is too powerful and it can fill up device buffer queue very fast.

I get best performance when I do busy wait, but this is not desirable
and is bad design.

I need to sleep and wake up quickly and predictability. Indication
from device that queue has space, is coming in form of memory write
(device writes to a memory location of i86 processor).

I tried using wait_event_interruptible_timeout, I am depending on 2nd
parameter of the function but it wake up is too slow, even tried using
value of 1.

Any suggestions ?

Sleep and Wake up

[Greg KH]

On Mon, May 02, 2011 at 11:32:32AM -0700, Abu Rasheda wrote:
> I am testing my driver on much faster host processor and facing
> following issues:
> 
> My host is too powerful and it can fill up device buffer queue very fast.

What kind of device is this?

> I get best performance when I do busy wait, but this is not desirable
> and is bad design.
> 
> I need to sleep and wake up quickly and predictability. Indication
> from device that queue has space, is coming in form of memory write
> (device writes to a memory location of i86 processor).
> 
> I tried using wait_event_interruptible_timeout, I am depending on 2nd
> parameter of the function but it wake up is too slow, even tried using
> value of 1.

Why not try increasing the buffer in your driver to handle any amount of
data needed?

thanks,

greg k-h

Sleep and Wake up

[Abu Rasheda]

On Mon, May 2, 2011 at 12:28 PM, Greg KH <greg@kroah.com> wrote:
> On Mon, May 02, 2011 at 11:32:32AM -0700, Abu Rasheda wrote:
>> I am testing my driver on much faster host processor and facing
>> following issues:
>>
>> My host is too powerful and it can fill up device buffer queue very fast.
>
> What kind of device is this?

its a networking device

>> I get best performance when I do busy wait, but this is not desirable
>> and is bad design.
>>
>> I need to sleep and wake up quickly and predictability. Indication
>> from device that queue has space, is coming in form of memory write
>> (device writes to a memory location of i86 processor).
>>
>> I tried using wait_event_interruptible_timeout, I am depending on 2nd
>> parameter of the function but it wake up is too slow, even tried using
>> value of 1.
>
> Why not try increasing the buffer in your driver to handle any amount of
> data needed?

Problem come from the fact that remote end can be slow, so device /
driver cannot do much. If I increase buffer size, than running on
(even) faster host processor will bring back same issue. I am looking
for solution which will scale.

I check for available buffer size if space is low, I sleep and retry.
This is causing lot of CPU to be wasted.

How can I optimize this ?

>
> thanks,
>
> greg k-h

Sleep and Wake up

[Greg KH]

On Mon, May 02, 2011 at 02:02:34PM -0700, Abu Rasheda wrote:
> On Mon, May 2, 2011 at 12:28 PM, Greg KH <greg@kroah.com> wrote:
> > On Mon, May 02, 2011 at 11:32:32AM -0700, Abu Rasheda wrote:
> >> I am testing my driver on much faster host processor and facing
> >> following issues:
> >>
> >> My host is too powerful and it can fill up device buffer queue very fast.
> >
> > What kind of device is this?
> 
> its a networking device

It sounds like a broken device, you need to be able to handle large data
streams, right?  Are you running Linux in this device and that is the
issue you are having?

> >> I get best performance when I do busy wait, but this is not desirable
> >> and is bad design.
> >>
> >> I need to sleep and wake up quickly and predictability. Indication
> >> from device that queue has space, is coming in form of memory write
> >> (device writes to a memory location of i86 processor).
> >>
> >> I tried using wait_event_interruptible_timeout, I am depending on 2nd
> >> parameter of the function but it wake up is too slow, even tried using
> >> value of 1.
> >
> > Why not try increasing the buffer in your driver to handle any amount of
> > data needed?
> 
> Problem come from the fact that remote end can be slow, so device /
> driver cannot do much. If I increase buffer size, than running on
> (even) faster host processor will bring back same issue. I am looking
> for solution which will scale.
> 
> I check for available buffer size if space is low, I sleep and retry.
> This is causing lot of CPU to be wasted.
> 
> How can I optimize this ?

Use NAPI for your network driver?

good luck,

greg k-h

Fwd: Sleep and Wake up

[Abu Rasheda]

> >> I am testing my driver on much faster host processor and facing
> >> following issues:
> >>
> >> My host is too powerful and it can fill up device buffer queue very fast.
> >
> > What kind of device is this?
>
>> its a networking device

>It sounds like a broken device, you need to be able to handle large data
>streams, right? ?Are you running Linux in this device and that is the
>issue you are having?

I am not running Linux on the device. As a said, if remote host does
not send syn for TCP packets, my device queue will continue to grow.

My question is simple, what would be correct way of handling things in
this situation in software or if device is broken how would device
behave differently ?

I am here to learn.

Kernel compilation

[Ezequiel García]

Greetings,After reading Linux Kernel in a Nutshell (by the way, great book), I now have my own kernel configured for my hardware needs.?
However, I have some really strange behavior. On tty1, the same tty that launches X11, I have to press twice each keyboard key.?Anyone has any idea what's going on? I am using a customized Debian Squeeze + OpenBox.
Thanks in advance, and thanks Greg for writing the book.
Ezequiel.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110502/4f88e012/attachment.html 

Kernel compilation

[Pico Geyer]

[Forgot to cc list]

2011/5/2 Ezequiel Garc?a <elezegarcia@yahoo.com.ar>
>
> Greetings,
> After reading Linux Kernel in a Nutshell (by the way, great book), I now have my own kernel configured for my hardware needs.
> However, I have some really strange behavior. On tty1, the same tty that launches X11, I have to press twice each keyboard key.?Anyone has any idea what's going on? I am using a customized Debian Squeeze + OpenBox.
> Thanks in advance, and thanks Greg for writing the book.
> Ezequiel.

Hi Ezequiel

I'm new on this list as well but I don't think anyone will be able to
help you with the limited info that you have provided.
When you say you have a kernel configured for your hardware, did you:

A) Download the kernel from http://www.kernel.org/ ? In that case you
need to tell us which version you're using.
B) Get the kernel from git a git tree http://git.kernel.org/? We still
need to know exactly which version
C) Recompile your debian kernel? (2.6.32?)

If this bug is present with only a change in kernel config, it would
be interesting to see the difference between the working and broken
config.

I hope that helps,
Pico

Sleep and Wake up

[Mulyadi Santosa]

Hi Abu...

On Tue, May 3, 2011 at 01:32, Abu Rasheda <rcpilot2010@gmail.com> wrote:
> I am testing my driver on much faster host processor and facing
> following issues:
>
> My host is too powerful and it can fill up device buffer queue very fast.
>
> I get best performance when I do busy wait, but this is not desirable
> and is bad design.

For me, busy waiting could be the best for your case... in fact, NAPI
sometimes do that too....in most cases.

Actually, devices usually fills the in-RAM buffer with the help of
interrupt(s). So whenever in-device buffer has something new, it
interrupts...then soft irq follows up by copying them to RAM buffe
(circular buffer usually). To make it fast, DMA  or other CPU less
operation kicks in here.

So I think that;s the key...by relying on the interrupts...then you
will know when your buffer is about to be full.

Other than that, I would still say, busy waiting could be your best bet.

PS: here's crazy idea, if you can accept it. Put a guard page, right
at the tail of your buffer,. Make it unwritable even by kernel mode.
Right when you hit it, page fault should kicks in. Then you know
buffer is full. This is inspired by the stack guard..usual method in
user space to reduce buffer overflow risk.

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

Sleep and Wake up

[Peter Teoh]

On Tue, May 3, 2011 at 2:32 AM, Abu Rasheda <rcpilot2010@gmail.com> wrote:

> I am testing my driver on much faster host processor and facing
> following issues:
>
> My host is too powerful and it can fill up device buffer queue very fast.
>
> I get best performance when I do busy wait, but this is not desirable
> and is bad design.
>
> I need to sleep and wake up quickly and predictability. Indication
> from device that queue has space, is coming in form of memory write
> (device writes to a memory location of i86 processor).
>
> I tried using wait_event_interruptible_timeout, I am depending on 2nd
> parameter of the function but it wake up is too slow, even tried using
> value of 1.
>
> Any suggestions ?
>
>
Many of us are just newbies in this area, and therefore, get it working is
much more important than to optimize it - u can safely said that the
hardcore kernel developer has already optimize many of these problems away,
and so if they cannot do it, there must be a reason...try to probe more
first perhaps.

High speed networking device has many special hardware features: IP/UDP/TCP
checksum offload etc.

http://www.fenrus.org/how-to-not-write-a-device-driver-paper.pdf
http://www.sun.com/products/networking/infiniband/ibhcaPCI-E/docs/datasheet.pdf

Read this about NAPI:

http://www.linuxfoundation.org/collaborate/workgroups/networking/napi

<http://www.linuxfoundation.org/collaborate/workgroups/networking/napi>Eg,
"Interrupt mitigation" - whereby interrupt mechanism is disabled (in
particular my desktop PC's r8196.c is using this feature) and polling takes
over instead - but u have to implemented complicated mechanism to reinject
the interrupt if necessary (read r8196.c).

And if u really ready....this is a good writeup:

http://datatag.web.cern.ch/datatag/howto/tcp.html

Other possible suggestion/features:

Jumbo frames:

http://www.cyberciti.biz/faq/rhel-centos-debian-ubuntu-jumbo-frames-configuration/

PCI posting (pdf paper above and r8196.c).

Disabling TCP software checksum (and use the hardware instead):

http://www.linuxquestions.org/questions/linux-enterprise-47/how-to-disable-tcp-checksumming-690745/

And for loads of other ideas (eg, TCP bypass):

*
http://ttthebear.blogspot.com/2008/07/linux-kernel-bypass-and-performance.html
*

Generally a lot of these ideas can be found in the kernel source codes -
just search and copy the implementation.....the highest network data
transfer is achieved in infiniband-based Mellanox card (in some China
supercomputer), and this involved the use of GPU technology etc....

https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2009-May/039639.html

-- 
Regards,
Peter Teoh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110510/4263ed73/attachment.html 

Sleep and Wake up

[Vikram Narayanan]

>> On Mon, May 02, 2011 at 11:32:32AM -0700, Abu Rasheda wrote:
>>> I am testing my driver on much faster host processor and facing
>>> following issues:
>>>
>>> My host is too powerful and it can fill up device buffer queue very fast.
>>
>> What kind of device is this?
>
> its a networking device
>
>>> I get best performance when I do busy wait, but this is not desirable
>>> and is bad design.
>>>
>>> I need to sleep and wake up quickly and predictability. Indication
>>> from device that queue has space, is coming in form of memory write
>>> (device writes to a memory location of i86 processor).
>>>
>>> I tried using wait_event_interruptible_timeout, I am depending on 2nd
>>> parameter of the function but it wake up is too slow, even tried using
>>> value of 1.
>>
>> Why not try increasing the buffer in your driver to handle any amount of
>> data needed?
>
> Problem come from the fact that remote end can be slow, so device /
> driver cannot do much. If I increase buffer size, than running on
> (even) faster host processor will bring back same issue. I am looking
> for solution which will scale.
AFAIK, These scaling things will be taken care by the protocol.
BTW, Which protocol you are communicating with?
It would be good if you try to give more information on what you are
trying to achieve.

Thanks,
Vikram