* Re: [uml-devel] Q: UML thread communication - scheduling oddness
2004-08-05 0:20 ` Jeff Dike
@ 2004-08-05 13:31 ` Michael Richardson
2004-08-05 13:35 ` Joe Marzot
2004-08-05 16:45 ` Joe Marzot
2 siblings, 0 replies; 7+ messages in thread
From: Michael Richardson @ 2004-08-05 13:31 UTC (permalink / raw)
To: user-mode-linux-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
>>>>> "Jeff" == Jeff Dike <jdike@addtoit.com> writes:
Jeff> gmarzot@nortelnetworks.com said:
>> That is, if I run a little busy loop /mnt/plankton/stress --cpu 1
>> --io 1 on the server side, the round trip time for messages is
>> greatly *improved*. without this activity and a basically dormant
>> server side UML the round trip times show considerable variabilty
>> with gaps on the order of seconds.
Jeff> How's this for a theory: The server's host is short on memory.
Jeff> A UML which is answering pings every second or so can be
The theory is sound to me.
The host doesn't have to be short on memory for this to happen either.
It suffices for there to be things that are groverling through the
file systems, and pulling in disk pages for processes to get swapped
out.
- --
] ON HUMILITY: to err is human. To moo, bovine. [
] Michael Richardson, Seaway Networks Corporation [
] michael@seawaynetworks.com http://www.seawaynetworks.com/ [
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Finger me for keys
iD8DBQFBEjaY22r3dfT9QZERArj+AKDsuVgBPeYb1goRZ0mB2ITUXrivswCgnJ5h
4sjmXFb/Y+wdPFpTgEnC+98=
=CjU1
-----END PGP SIGNATURE-----
-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [uml-devel] Q: UML thread communication - scheduling oddness
2004-08-05 0:20 ` Jeff Dike
2004-08-05 13:31 ` Michael Richardson
@ 2004-08-05 13:35 ` Joe Marzot
2004-08-05 17:23 ` Jeff Dike
2004-08-05 16:45 ` Joe Marzot
2 siblings, 1 reply; 7+ messages in thread
From: Joe Marzot @ 2004-08-05 13:35 UTC (permalink / raw)
To: Jeff Dike; +Cc: Joe Marzot, user-mode-linux-devel, Randy Macleod
That's an excellent theory esp. ince we do not see the problem typically
on a host with more memory. I presume there are tricks we can play to
get our memory not to be swapped out (recall a discussion on that here).
Could you add a few sentences though on the delivery mechnisms for the
the packets to the kernel thread and to userspace. I saw a poll() on the
pollfds array which has the taps fd in it but this is a non-blocking
poll and I could not trace to how often it is called and what triggers
it - I also saw the sigio handler but I could not see who was generating
the SIGIO - is that how the userspace thread is woken up?
If there is some thing else I can read to get some background that would
help...otherwise its back to the source of course.
thank you kindly, Giovanni
Jeff Dike wrote:
> gmarzot@nortelnetworks.com said:
> > That is, if I run a little busy loop
> > /mnt/plankton/stress --cpu 1 --io 1
> > on the server side, the round trip time for messages is greatly
> > *improved*. without this activity and a basically dormant server side
> > UML the round trip times show considerable variabilty with gaps on the
> > order of seconds.
>
> How's this for a theory: The server's host is short on memory. A UML
> which
> is answering pings every second or so can be considered pretty idle, and
> thus
> a good candidate to have its pages swapped out. So, this happens, and when
> a ping comes in, UML needs to be swapped back in, causing your long and
> unpredictable delays.
>
> Running a busy loop in the UML makes it busy from the point of view of the
> host and prevents it from being swapped out, giving you nice ping
> latencies.
>
> Jeff
>
>
-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [uml-devel] Q: UML thread communication - scheduling oddness
2004-08-05 13:35 ` Joe Marzot
@ 2004-08-05 17:23 ` Jeff Dike
0 siblings, 0 replies; 7+ messages in thread
From: Jeff Dike @ 2004-08-05 17:23 UTC (permalink / raw)
To: Joe Marzot; +Cc: user-mode-linux-devel, Randy Macleod
gmarzot@nortelnetworks.com said:
> Could you add a few sentences though on the delivery mechnisms for the
> the packets to the kernel thread and to userspace. I saw a poll() on
> the pollfds array which has the taps fd in it but this is a
> non-blocking poll and I could not trace to how often it is called and
> what triggers it - I also saw the sigio handler but I could not see
> who was generating the SIGIO - is that how the userspace thread is
> woken up?
OK, an incoming packet is handled this way:
it's received by the host and routed to the tap device (or
alternatively, it's received by the host eth, sent to the bridge that it's part
of, and copied to the tap device which is also in the bridge)
it pops out of /dev/net/tun, which UML has a file descriptor open to
the file descriptor is set up O_ASYNC, so that generates a SIGIO
this causes sigio_handler to be called
it looks at all of the descriptors in its care, and handles all the
ones that are now active
the network driver's interrupt routine is called whenever a tap fd is
active, and it reads the frame off the fd and passes it to the UML network
stack
sigio_handler does blocking polls because it knows that something is active,
and that it won't, in fact, block.
Jeff
-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [uml-devel] Q: UML thread communication - scheduling oddness
2004-08-05 0:20 ` Jeff Dike
2004-08-05 13:31 ` Michael Richardson
2004-08-05 13:35 ` Joe Marzot
@ 2004-08-05 16:45 ` Joe Marzot
2004-08-05 19:17 ` Jeff Dike
2 siblings, 1 reply; 7+ messages in thread
From: Joe Marzot @ 2004-08-05 16:45 UTC (permalink / raw)
To: Jeff Dike; +Cc: Joe Marzot, user-mode-linux-devel, Randy Macleod
Jeff Dike wrote:
> gmarzot@nortelnetworks.com said:
> > That is, if I run a little busy loop
> > /mnt/plankton/stress --cpu 1 --io 1
> > on the server side, the round trip time for messages is greatly
> > *improved*. without this activity and a basically dormant server side
> > UML the round trip times show considerable variabilty with gaps on the
> > order of seconds.
>
> How's this for a theory: The server's host is short on memory. A UML
> which
> is answering pings every second or so can be considered pretty idle, and
> thus
> a good candidate to have its pages swapped out. So, this happens, and when
> a ping comes in, UML needs to be swapped back in, causing your long and
> unpredictable delays.
thinking about this a bit more...some doubts...note: the regular 'pings'
which are serviced purely by kernel resident code/ kernel thread do not
exhibit the delays. It is only when the communication involved a
userspace app as the responder to the 'app-level-ping' that the delays
come. Is the memory for the userspace app seperately swappable from that
of the kernel thread? If so then the theory still makes sense, since the
kernel thread likely has more going on so that is less likely to swap.
If not, I am still confused.
-GSM
>
> Running a busy loop in the UML makes it busy from the point of view of the
> host and prevents it from being swapped out, giving you nice ping
> latencies.
>
> Jeff
>
>
-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [uml-devel] Q: UML thread communication - scheduling oddness
2004-08-05 16:45 ` Joe Marzot
@ 2004-08-05 19:17 ` Jeff Dike
0 siblings, 0 replies; 7+ messages in thread
From: Jeff Dike @ 2004-08-05 19:17 UTC (permalink / raw)
To: Joe Marzot; +Cc: user-mode-linux-devel, Randy Macleod
gmarzot@nortelnetworks.com said:
> thinking about this a bit more...some doubts...note: the regular
> 'pings' which are serviced purely by kernel resident code/ kernel
> thread do not exhibit the delays. It is only when the communication
> involved a userspace app as the responder to the 'app-level-ping'
> that the delays come. Is the memory for the userspace app seperately
> swappable from that of the kernel thread?
The userspace app is all of UML, including the parts of the kernel which
respond to pings. It can all be swapped out by the host.
Jeff
-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
^ permalink raw reply [flat|nested] 7+ messages in thread