Lost events in older kernels

Lost events in older kernels

[Rafi Rubin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm playing with a project with a 2.6.29 kernel, and the userspace application
seems to miss some events.  Is there a particular fix that improved the handling?

Also tried catting the dev to a file while testing, and the dump also is missing
some events.

Rafi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkv3glkACgkQwuRiAT9o609vZACffkFSeTCDcNONThELFmUNx7Rk
LbQAn1Zzw0w1TcRn16hC8mBgpipF6ZL8
=F24i
-----END PGP SIGNATURE-----

Re: Lost events in older kernels

[Dmitry Torokhov]

On Sat, May 22, 2010 at 03:06:07AM -0400, Rafi Rubin wrote:
> I'm playing with a project with a 2.6.29 kernel, and the userspace application
> seems to miss some events.  Is there a particular fix that improved the handling?
> 
> Also tried catting the dev to a file while testing, and the dump also is missing
> some events.
>

No "interesting" patches went into evdev for a few release now...

Hm, could it be that event queue is overflowing before userspace gets a
chance to empty it. What kind of event rate are we talking here?

-- 
Dmitry

Re: Lost events in older kernels

[Rafi Rubin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/22/10 03:42, Dmitry Torokhov wrote:
> On Sat, May 22, 2010 at 03:06:07AM -0400, Rafi Rubin wrote:
>> I'm playing with a project with a 2.6.29 kernel, and the userspace application
>> seems to miss some events.  Is there a particular fix that improved the handling?
>>
>> Also tried catting the dev to a file while testing, and the dump also is missing
>> some events.
>>
> 
> No "interesting" patches went into evdev for a few release now...
> 
> Hm, could it be that event queue is overflowing before userspace gets a
> chance to empty it. What kind of event rate are we talking here?
> 

Quite possibly.  It is a multitouch device and we know Henrik's been concerned
with the load for a while.

So to put some numbers behind his fears:

146668 hid events processed
24952 evdev events captured with a cat
30 seconds (give or take).

This is for a mix of different numbers of fingers, but continuous use for those
30 seconds.  And X was running and reading the dev node too.

Rafi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkv3okcACgkQwuRiAT9o609A7ACg44p+YjeGhsDPZ3nEsWAscWK6
6akAoPp/Vx1GjGCtQCfvPNTNzQHgAdTA
=m4Mt
-----END PGP SIGNATURE-----

Re: Lost events in older kernels

[Henrik Rydberg]

Rafi Rubin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 05/22/10 03:42, Dmitry Torokhov wrote:
>> On Sat, May 22, 2010 at 03:06:07AM -0400, Rafi Rubin wrote:
>>> I'm playing with a project with a 2.6.29 kernel, and the userspace application
>>> seems to miss some events.  Is there a particular fix that improved the handling?
>>>
>>> Also tried catting the dev to a file while testing, and the dump also is missing
>>> some events.
>>>
>> No "interesting" patches went into evdev for a few release now...
>>
>> Hm, could it be that event queue is overflowing before userspace gets a
>> chance to empty it. What kind of event rate are we talking here?
>>
> 
> Quite possibly.  It is a multitouch device and we know Henrik's been concerned
> with the load for a while.
> 
> So to put some numbers behind his fears:
> 
> 146668 hid events processed
> 24952 evdev events captured with a cat
> 30 seconds (give or take).
> 
> This is for a mix of different numbers of fingers, but continuous use for those
> 30 seconds.  And X was running and reading the dev node too.

Others have experienced this too. Mika has a patch for this, increasing the
(kernel) evdev buffer quite a bit, from 64 to 256. I believe the reason it is
not sent upstream is because it increases the footprint by 3 Kb. Perhaps a
dynamic solution could work here.

Henrik

Re: Lost events in older kernels

[Dmitry Torokhov]

On Sat, May 22, 2010 at 12:27:15PM +0200, Henrik Rydberg wrote:
> Rafi Rubin wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> > 
> > On 05/22/10 03:42, Dmitry Torokhov wrote:
> >> On Sat, May 22, 2010 at 03:06:07AM -0400, Rafi Rubin wrote:
> >>> I'm playing with a project with a 2.6.29 kernel, and the userspace application
> >>> seems to miss some events.  Is there a particular fix that improved the handling?
> >>>
> >>> Also tried catting the dev to a file while testing, and the dump also is missing
> >>> some events.
> >>>
> >> No "interesting" patches went into evdev for a few release now...
> >>
> >> Hm, could it be that event queue is overflowing before userspace gets a
> >> chance to empty it. What kind of event rate are we talking here?
> >>
> > 
> > Quite possibly.  It is a multitouch device and we know Henrik's been concerned
> > with the load for a while.
> > 
> > So to put some numbers behind his fears:
> > 
> > 146668 hid events processed
> > 24952 evdev events captured with a cat
> > 30 seconds (give or take).
> > 
> > This is for a mix of different numbers of fingers, but continuous use for those
> > 30 seconds.  And X was running and reading the dev node too.
> 
> Others have experienced this too. Mika has a patch for this, increasing the
> (kernel) evdev buffer quite a bit, from 64 to 256. I believe the reason it is
> not sent upstream is because it increases the footprint by 3 Kb. Perhaps a
> dynamic solution could work here.
> 

Yes, if input devices could hint handlers about their "packet size"
then evdev could size it's event queue accordingly. I'd say we need to
keep about 8 packets worth of data (number is pulled right out of my
behind ;) )...

-- 
Dmitry

Re: Lost events in older kernels

[Henrik Rydberg]

Dmitry Torokhov wrote:
> On Sat, May 22, 2010 at 12:27:15PM +0200, Henrik Rydberg wrote:
>> Rafi Rubin wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> On 05/22/10 03:42, Dmitry Torokhov wrote:
>>>> On Sat, May 22, 2010 at 03:06:07AM -0400, Rafi Rubin wrote:
>>>>> I'm playing with a project with a 2.6.29 kernel, and the userspace application
>>>>> seems to miss some events.  Is there a particular fix that improved the handling?
>>>>>
>>>>> Also tried catting the dev to a file while testing, and the dump also is missing
>>>>> some events.
>>>>>
>>>> No "interesting" patches went into evdev for a few release now...
>>>>
>>>> Hm, could it be that event queue is overflowing before userspace gets a
>>>> chance to empty it. What kind of event rate are we talking here?
>>>>
>>> Quite possibly.  It is a multitouch device and we know Henrik's been concerned
>>> with the load for a while.
>>>
>>> So to put some numbers behind his fears:
>>>
>>> 146668 hid events processed
>>> 24952 evdev events captured with a cat
>>> 30 seconds (give or take).

Thanks for these numbers, Rafi.

>>> This is for a mix of different numbers of fingers, but continuous use for those
>>> 30 seconds.  And X was running and reading the dev node too.
>> Others have experienced this too. Mika has a patch for this, increasing the
>> (kernel) evdev buffer quite a bit, from 64 to 256. I believe the reason it is
>> not sent upstream is because it increases the footprint by 3 Kb. Perhaps a
>> dynamic solution could work here.
>>
> 
> Yes, if input devices could hint handlers about their "packet size"
> then evdev could size it's event queue accordingly. I'd say we need to
> keep about 8 packets worth of data (number is pulled right out of my
> behind ;) )...
> 

Yeah. The bcm5974 driver sends 8 events per finger, plus some ST stuff. With
four fingers on the pad, that amounts to roughly 40 events per packet.

146668 / 24952 = 5.9
40 * 8 / 64 = 5.0

Looks reasonable, doesn't it?

Regarding the packet size hinting, the handler already knows which events are
potentially being sent, and could produce a reasonable buffer size from it. In
particular if it knows which events are bypassing filtering. ;-)

Henrik

Re: Lost events in older kernels

[Dmitry Torokhov]

On Sat, May 22, 2010 at 10:57:42PM +0200, Henrik Rydberg wrote:
> Dmitry Torokhov wrote:
> > On Sat, May 22, 2010 at 12:27:15PM +0200, Henrik Rydberg wrote:
> >> Rafi Rubin wrote:
> >>> -----BEGIN PGP SIGNED MESSAGE-----
> >>> Hash: SHA1
> >>>
> >>> On 05/22/10 03:42, Dmitry Torokhov wrote:
> >>>> On Sat, May 22, 2010 at 03:06:07AM -0400, Rafi Rubin wrote:
> >>>>> I'm playing with a project with a 2.6.29 kernel, and the userspace application
> >>>>> seems to miss some events.  Is there a particular fix that improved the handling?
> >>>>>
> >>>>> Also tried catting the dev to a file while testing, and the dump also is missing
> >>>>> some events.
> >>>>>
> >>>> No "interesting" patches went into evdev for a few release now...
> >>>>
> >>>> Hm, could it be that event queue is overflowing before userspace gets a
> >>>> chance to empty it. What kind of event rate are we talking here?
> >>>>
> >>> Quite possibly.  It is a multitouch device and we know Henrik's been concerned
> >>> with the load for a while.
> >>>
> >>> So to put some numbers behind his fears:
> >>>
> >>> 146668 hid events processed
> >>> 24952 evdev events captured with a cat
> >>> 30 seconds (give or take).
> 
> Thanks for these numbers, Rafi.
> 
> >>> This is for a mix of different numbers of fingers, but continuous use for those
> >>> 30 seconds.  And X was running and reading the dev node too.
> >> Others have experienced this too. Mika has a patch for this, increasing the
> >> (kernel) evdev buffer quite a bit, from 64 to 256. I believe the reason it is
> >> not sent upstream is because it increases the footprint by 3 Kb. Perhaps a
> >> dynamic solution could work here.
> >>
> > 
> > Yes, if input devices could hint handlers about their "packet size"
> > then evdev could size it's event queue accordingly. I'd say we need to
> > keep about 8 packets worth of data (number is pulled right out of my
> > behind ;) )...
> > 
> 
> Yeah. The bcm5974 driver sends 8 events per finger, plus some ST stuff. With
> four fingers on the pad, that amounts to roughly 40 events per packet.
> 
> 146668 / 24952 = 5.9
> 40 * 8 / 64 = 5.0
> 
> Looks reasonable, doesn't it?
> 
> Regarding the packet size hinting, the handler already knows which events are
> potentially being sent, and could produce a reasonable buffer size from it. In
> particular if it knows which events are bypassing filtering. ;-)
> 

Yes, but the driver knows for sure. Why making one guess when another
can tell?

-- 
Dmitry

Re: Lost events in older kernels

[Henrik Rydberg]

Dmitry Torokhov wrote:
> On Sat, May 22, 2010 at 10:57:42PM +0200, Henrik Rydberg wrote:
>> Dmitry Torokhov wrote:
>>> On Sat, May 22, 2010 at 12:27:15PM +0200, Henrik Rydberg wrote:
>>>> Rafi Rubin wrote:
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA1
>>>>>
>>>>> On 05/22/10 03:42, Dmitry Torokhov wrote:
>>>>>> On Sat, May 22, 2010 at 03:06:07AM -0400, Rafi Rubin wrote:
>>>>>>> I'm playing with a project with a 2.6.29 kernel, and the userspace application
>>>>>>> seems to miss some events.  Is there a particular fix that improved the handling?
>>>>>>>
>>>>>>> Also tried catting the dev to a file while testing, and the dump also is missing
>>>>>>> some events.
>>>>>>>
>>>>>> No "interesting" patches went into evdev for a few release now...
>>>>>>
>>>>>> Hm, could it be that event queue is overflowing before userspace gets a
>>>>>> chance to empty it. What kind of event rate are we talking here?
>>>>>>
>>>>> Quite possibly.  It is a multitouch device and we know Henrik's been concerned
>>>>> with the load for a while.
>>>>>
>>>>> So to put some numbers behind his fears:
>>>>>
>>>>> 146668 hid events processed
>>>>> 24952 evdev events captured with a cat
>>>>> 30 seconds (give or take).
>> Thanks for these numbers, Rafi.
>>
>>>>> This is for a mix of different numbers of fingers, but continuous use for those
>>>>> 30 seconds.  And X was running and reading the dev node too.
>>>> Others have experienced this too. Mika has a patch for this, increasing the
>>>> (kernel) evdev buffer quite a bit, from 64 to 256. I believe the reason it is
>>>> not sent upstream is because it increases the footprint by 3 Kb. Perhaps a
>>>> dynamic solution could work here.
>>>>
>>> Yes, if input devices could hint handlers about their "packet size"
>>> then evdev could size it's event queue accordingly. I'd say we need to
>>> keep about 8 packets worth of data (number is pulled right out of my
>>> behind ;) )...
>>>
>> Yeah. The bcm5974 driver sends 8 events per finger, plus some ST stuff. With
>> four fingers on the pad, that amounts to roughly 40 events per packet.
>>
>> 146668 / 24952 = 5.9
>> 40 * 8 / 64 = 5.0
>>
>> Looks reasonable, doesn't it?
>>
>> Regarding the packet size hinting, the handler already knows which events are
>> potentially being sent, and could produce a reasonable buffer size from it. In
>> particular if it knows which events are bypassing filtering. ;-)
>>
> 
> Yes, but the driver knows for sure. Why making one guess when another
> can tell?
> 

I was thinking that the data stream itself should be enough to dynamically
adjust the buffer size, given an initial appropriate size. In theory, that could
even be better than asking the driver. It would also avoid changing all drivers.
Just a thought. Either way, the problem seems real.

Henrik

Re: Lost events in older kernels

[Rafi Rubin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/22/10 16:12, Dmitry Torokhov wrote:
> On Sat, May 22, 2010 at 12:27:15PM +0200, Henrik Rydberg wrote:
>> Rafi Rubin wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> On 05/22/10 03:42, Dmitry Torokhov wrote:
>>>> On Sat, May 22, 2010 at 03:06:07AM -0400, Rafi Rubin wrote:
>>>>> I'm playing with a project with a 2.6.29 kernel, and the userspace application
>>>>> seems to miss some events.  Is there a particular fix that improved the handling?
>>>>>
>>>>> Also tried catting the dev to a file while testing, and the dump also is missing
>>>>> some events.
>>>>>
>>>> No "interesting" patches went into evdev for a few release now...
>>>>
>>>> Hm, could it be that event queue is overflowing before userspace gets a
>>>> chance to empty it. What kind of event rate are we talking here?
>>>>
>>>
>>> Quite possibly.  It is a multitouch device and we know Henrik's been concerned
>>> with the load for a while.
>>>
>>> So to put some numbers behind his fears:
>>>
>>> 146668 hid events processed
>>> 24952 evdev events captured with a cat
>>> 30 seconds (give or take).
>>>
>>> This is for a mix of different numbers of fingers, but continuous use for those
>>> 30 seconds.  And X was running and reading the dev node too.
>>
>> Others have experienced this too. Mika has a patch for this, increasing the
>> (kernel) evdev buffer quite a bit, from 64 to 256. I believe the reason it is
>> not sent upstream is because it increases the footprint by 3 Kb. Perhaps a
>> dynamic solution could work here.
>>
> 
> Yes, if input devices could hint handlers about their "packet size"
> then evdev could size it's event queue accordingly. I'd say we need to
> keep about 8 packets worth of data (number is pulled right out of my
> behind ;) )...

Um, turns out I missed input_init_abs_bypass(void) in my backport, and for some
reason get a bit of corruption ;)

Sorry for the false alarm.

As for buffering, strategies, if we have a rate of reports from the device, I'd
vote for sizing the buffer based on time.

Does the code make sure we don't drop parts of a report?

Rafi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkv4jp8ACgkQwuRiAT9o609TnwCfbEPVG5igpiy8w4P3HiRg9ecV
Y7gAmwW97yX2ElVepFIIS3Fm6SiUz9du
=+mW7
-----END PGP SIGNATURE-----

Re: Lost events in older kernels

[Dmitry Torokhov]

On Sat, May 22, 2010 at 10:10:42PM -0400, Rafi Rubin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 05/22/10 16:12, Dmitry Torokhov wrote:
> > On Sat, May 22, 2010 at 12:27:15PM +0200, Henrik Rydberg wrote:
> >> Rafi Rubin wrote:
> >>> -----BEGIN PGP SIGNED MESSAGE-----
> >>> Hash: SHA1
> >>>
> >>> On 05/22/10 03:42, Dmitry Torokhov wrote:
> >>>> On Sat, May 22, 2010 at 03:06:07AM -0400, Rafi Rubin wrote:
> >>>>> I'm playing with a project with a 2.6.29 kernel, and the userspace application
> >>>>> seems to miss some events.  Is there a particular fix that improved the handling?
> >>>>>
> >>>>> Also tried catting the dev to a file while testing, and the dump also is missing
> >>>>> some events.
> >>>>>
> >>>> No "interesting" patches went into evdev for a few release now...
> >>>>
> >>>> Hm, could it be that event queue is overflowing before userspace gets a
> >>>> chance to empty it. What kind of event rate are we talking here?
> >>>>
> >>>
> >>> Quite possibly.  It is a multitouch device and we know Henrik's been concerned
> >>> with the load for a while.
> >>>
> >>> So to put some numbers behind his fears:
> >>>
> >>> 146668 hid events processed
> >>> 24952 evdev events captured with a cat
> >>> 30 seconds (give or take).
> >>>
> >>> This is for a mix of different numbers of fingers, but continuous use for those
> >>> 30 seconds.  And X was running and reading the dev node too.
> >>
> >> Others have experienced this too. Mika has a patch for this, increasing the
> >> (kernel) evdev buffer quite a bit, from 64 to 256. I believe the reason it is
> >> not sent upstream is because it increases the footprint by 3 Kb. Perhaps a
> >> dynamic solution could work here.
> >>
> > 
> > Yes, if input devices could hint handlers about their "packet size"
> > then evdev could size it's event queue accordingly. I'd say we need to
> > keep about 8 packets worth of data (number is pulled right out of my
> > behind ;) )...
> 
> Um, turns out I missed input_init_abs_bypass(void) in my backport, and for some
> reason get a bit of corruption ;)
> 
> Sorry for the false alarm.
> 
> As for buffering, strategies, if we have a rate of reports from the device, I'd
> vote for sizing the buffer based on time.
> 
> Does the code make sure we don't drop parts of a report?
> 

No, it does not. Evdev has no idea about the size of a report.

-- 
Dmitry

Re: Lost events in older kernels

[Henrik Rydberg]

Rafi Rubin wrote:
[...]
> Um, turns out I missed input_init_abs_bypass(void) in my backport, and for some
> reason get a bit of corruption ;)
> 
> Sorry for the false alarm.

If your code executed without bypassing event filtering, it would only mean less
events were sent, which would not void the argument. If you have updated
numbers, it would be very interesting to see them.

Henrik

Re: Lost events in older kernels

[Dmitry Torokhov]

On Sunday 23 May 2010 01:44:25 am Henrik Rydberg wrote:
> Rafi Rubin wrote:
> [...]
> 
> > Um, turns out I missed input_init_abs_bypass(void) in my backport, and
> > for some reason get a bit of corruption ;)
> > 
> > Sorry for the false alarm.
> 
> If your code executed without bypassing event filtering, it would only mean
> less events were sent,

Right, and that would look like some events (multitouch ones) were dropped,
whereas they were simply filtered out. 

> which would not void the argument. If you have
> updated numbers, it would be very interesting to see them.

Yes, we still need to think about [re]sizing event queue, especially for
multi-touch devices.

-- 
Dmitry