Good news for threading!

Good news for threading!

[Alan Jenkins]

I solved one of the reservations that held me back on threaded udevd.  I 
still have some other concerns to work on, but I'm feeling much more 
optimistic now.

Replacing processes with threads reduced page faults (copy-on-write) by 
60-80%, but that _still_ left over 10% of udevd time in page fault overhead.

I've discovered I can reduce this overhead much further by marking 
thread stacks with MADV_DONTFORK.  When a thread needs to fork an 
external program, it can temporarily unmark its own thread stack.

Disclaimer: I have no idea why this should reduce the number of page 
faults.  I could be getting something horribly wrong.  But even if it's 
wrong, at least it gives me a new angle on this problem.

Alan

Re: Good news for threading!

[Kay Sievers]

On Mon, May 25, 2009 at 15:46, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
> I solved one of the reservations that held me back on threaded udevd.  I
> still have some other concerns to work on, but I'm feeling much more
> optimistic now.
>
> Replacing processes with threads reduced page faults (copy-on-write) by
> 60-80%, but that _still_ left over 10% of udevd time in page fault overhead.
>
> I've discovered I can reduce this overhead much further by marking thread
> stacks with MADV_DONTFORK.  When a thread needs to fork an external program,
> it can temporarily unmark its own thread stack.
>
> Disclaimer: I have no idea why this should reduce the number of page faults.
>  I could be getting something horribly wrong.  But even if it's wrong, at
> least it gives me a new angle on this problem.

Do you have numbers for the difference of threaded vs. non-threaded, for:
  time (udevadm trigger; udevadm settle)
when no rules are active?

On my box it's ~0.6 seconds for the non-threaded udevd and ~460
devices, when I move all rules out of the way.

Thanks,
Kay

Re: Good news for threading!

[Alan Jenkins]

Kay Sievers wrote:
> On Mon, May 25, 2009 at 15:46, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
>   
>> I solved one of the reservations that held me back on threaded udevd.  I
>> still have some other concerns to work on, but I'm feeling much more
>> optimistic now.
>>
>> Replacing processes with threads reduced page faults (copy-on-write) by
>> 60-80%, but that _still_ left over 10% of udevd time in page fault overhead.
>>
>> I've discovered I can reduce this overhead much further by marking thread
>> stacks with MADV_DONTFORK.  When a thread needs to fork an external program,
>> it can temporarily unmark its own thread stack.
>>
>> Disclaimer: I have no idea why this should reduce the number of page faults.
>>  I could be getting something horribly wrong.  But even if it's wrong, at
>> least it gives me a new angle on this problem.
>>     
>
> Do you have numbers for the difference of threaded vs. non-threaded, for:
>   time (udevadm trigger; udevadm settle)
> when no rules are active?
>
> On my box it's ~0.6 seconds for the non-threaded udevd and ~460
> devices, when I move all rules out of the way.
>
> Thanks,
> Kay
>   

Interesting question.  I stop HAL first, it tends to slow my tests down 
and doesn't represent normal bootup.

Test machine is eeepc 701; trigger -n shows 382 devices.

no threading:    ~0.5s
threading:   ~0.25s

This is based on an a slightly out of date tree.  Your latest HEAD is 
still ~0.5s, but perhaps slightly lower, 0.46-0.48 as opposed to 0.48-0.5.

Alan

Re: Good news for threading!

[Kay Sievers]

On Mon, May 25, 2009 at 22:28, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
> Kay Sievers wrote:

>> Do you have numbers for the difference of threaded vs. non-threaded, for:
>>  time (udevadm trigger; udevadm settle)
>> when no rules are active?

> Interesting question.  I stop HAL first, it tends to slow my tests down and
> doesn't represent normal bootup.

Yeah, it's the RUN+="socket: ..." handling, which can block the event
process, if something listens, but does not read fast enough. With the
new netlink events from udev to listeners (no RUN+="socket: ..."
needed), the sending is de-coupled from the listeners state and will
not slow-down udev. Hopefully HAL will go away soon, we are pretty
close already.

> Test machine is eeepc 701; trigger -n shows 382 devices.
>
> no threading:    ~0.5s
> threading:   ~0.25s

Oh, nice numbers. In a real setup, can we expect more than 0.25s to
win with a threaded udevd? What can we expect from the udev-exec
thing? Will it be a bit slower, because it has a bit more work to do?

Thanks,
Kay

Re: Good news for threading!

[Scott James Remnant]

[-- Attachment #1: Type: text/plain, Size: 380 bytes --]

On Mon, 2009-05-25 at 14:46 +0100, Alan Jenkins wrote:

> I solved one of the reservations that held me back on threaded udevd.  I 
> still have some other concerns to work on, but I'm feeling much more 
> optimistic now.
> 
Are you maintaining a GIT tree of your threading work?  I'd be very
interested in playing <g>

Scott
-- 
Scott James Remnant
scott@ubuntu.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Good news for threading!

[Alan Jenkins]

Kay Sievers wrote:
> On Mon, May 25, 2009 at 22:28, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
>   
>> Kay Sievers wrote:
>>     
>
>   
>>> Do you have numbers for the difference of threaded vs. non-threaded, for:
>>>  time (udevadm trigger; udevadm settle)
>>> when no rules are active?
>>>       
>
>   
>> Interesting question.  I stop HAL first, it tends to slow my tests down and
>> doesn't represent normal bootup.
>>     
>
> Yeah, it's the RUN+="socket: ..." handling, which can block the event
> process, if something listens, but does not read fast enough. With the
> new netlink events from udev to listeners (no RUN+="socket: ..."
> needed), the sending is de-coupled from the listeners state and will
> not slow-down udev. Hopefully HAL will go away soon, we are pretty
> close already.
>   

From my POV it's more that if I don't kill HAL, I think it appears
higher than udev in the profile.  So udev will have to compete for CPU
time, and I can't simply measure the elapsed time.

>> Test machine is eeepc 701; trigger -n shows 382 devices.
>>
>> no threading:    ~0.5s
>> threading:   ~0.25s
>>     
>
> Oh, nice numbers. In a real setup, can we expect more than 0.25s to
> win with a threaded udevd?

I don't know yet.  I need to re-implement the madvise hack so it doesn't
segfault on a booting system.

I'm pretty sure I know what I did wrong.  But right now I'm working on
submitting everything that's ready.

> What can we expect from the udev-exec
> thing? Will it be a bit slower, because it has a bit more work to do?
>   

I ditched the udev-exec daemon on Scotts suggestion.

Regards
Alan

Re: Good news for threading!

[Alan Jenkins]

Scott James Remnant wrote:
> On Mon, 2009-05-25 at 14:46 +0100, Alan Jenkins wrote:
>
>   
>> I solved one of the reservations that held me back on threaded udevd.  I 
>> still have some other concerns to work on, but I'm feeling much more 
>> optimistic now.
>>
>>     
> Are you maintaining a GIT tree of your threading work?  I'd be very
> interested in playing <g>
>
> Scott
>   

No, sorry.  I'm going to try to submit it all as patches Right Now.  I
will need to rewrite the madvise hack before I can submit that though.

Alan