[uml-devel] Q: UML thread communication - scheduling oddness

[uml-devel] Q: UML thread communication - scheduling oddness

[Joe Marzot]

We are seeing some sub-optimal behaviour (at least for our use) for 
communication between several UMLs. It appears the responsiveness of a 
given UML to application level network messaging is a function of how 
busy the serverside UML is.

That is, if I run a little busy loop

  /mnt/plankton/stress --cpu 1 --io 1

on the server side, the round trip time for messages is greatly 
*improved*. without this activity and a basically dormant server side 
UML the round trip times show considerable variabilty with gaps on the 
order of seconds.

Now for the question:

Can some one explain the event mechanism and communication pathways for 
network packets(using tuntap networking) delivered to the UML kernel and 
then for the kernel to a userspace process?

It looks like the kernel thread is responsive enough because if we do 
simple pings they don't suffer from the huge gaps as the user based 
pings do. I would like to understand how the kernel thread wakes up to 
grab the packets from the tap device and then how it notifies the user 
space thread to deal with it...ultimately maybe there is something we 
can do to make the userspace thread have less latency (other than 
running my dummy busy thing from above of course).

thanks for any help in understanding.

regards, Giovanni

ps. We are using 2.4.2x with 2.4.24-1 patch in skas mode



-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Q: UML thread communication - scheduling oddness

[Jeff Dike]

gmarzot@nortelnetworks.com said:
> That is, if I run a little busy loop
>   /mnt/plankton/stress --cpu 1 --io 1
> on the server side, the round trip time for messages is greatly
> *improved*. without this activity and a basically dormant server side
> UML the round trip times show considerable variabilty with gaps on the
>  order of seconds. 

How's this for a theory:  The server's host is short on memory.  A UML which
is answering pings every second or so can be considered pretty idle, and thus
a good candidate to have its pages swapped out.  So, this happens, and when
a ping comes in, UML needs to be swapped back in, causing your long and
unpredictable delays.

Running a busy loop in the UML makes it busy from the point of view of the
host and prevents it from being swapped out, giving you nice ping latencies.

				Jeff



-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Q: UML thread communication - scheduling oddness

[Michael Richardson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>>>>> "Jeff" == Jeff Dike <jdike@addtoit.com> writes:

    Jeff> gmarzot@nortelnetworks.com said:
    >> That is, if I run a little busy loop /mnt/plankton/stress --cpu 1
    >> --io 1 on the server side, the round trip time for messages is
    >> greatly *improved*. without this activity and a basically dormant
    >> server side UML the round trip times show considerable variabilty
    >> with gaps on the order of seconds.

    Jeff> How's this for a theory: The server's host is short on memory.
    Jeff> A UML which is answering pings every second or so can be

  The theory is sound to me.

  The host doesn't have to be short on memory for this to happen either.

  It suffices for there to be things that are groverling through the
file systems, and pulling in disk pages for processes to get swapped
out.

- --
]       ON HUMILITY: to err is human. To moo, bovine.                         [
]   Michael Richardson,            Seaway Networks Corporation                [
]   michael@seawaynetworks.com     http://www.seawaynetworks.com/             [
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Finger me for keys

iD8DBQFBEjaY22r3dfT9QZERArj+AKDsuVgBPeYb1goRZ0mB2ITUXrivswCgnJ5h
4sjmXFb/Y+wdPFpTgEnC+98=
=CjU1
-----END PGP SIGNATURE-----


-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Q: UML thread communication - scheduling oddness

[Joe Marzot]

That's an excellent theory esp. ince we do not see the problem typically 
on a host with more memory. I presume there are tricks we can play to 
get our memory not to be swapped out (recall a discussion on that here).

Could you add a few sentences though on the delivery mechnisms for the 
the packets to the kernel thread and to userspace. I saw a poll() on the 
pollfds array which has the taps fd in it but this is a non-blocking 
poll and I could not trace to how often it is called and what triggers 
it - I also saw the sigio handler but I could not see who was generating 
the SIGIO - is that how the userspace thread is woken up?

If there is some thing else I can read to get some background that would 
help...otherwise its back to the source of course.

thank you kindly, Giovanni

Jeff Dike wrote:
> gmarzot@nortelnetworks.com said:
>  > That is, if I run a little busy loop
>  >   /mnt/plankton/stress --cpu 1 --io 1
>  > on the server side, the round trip time for messages is greatly
>  > *improved*. without this activity and a basically dormant server side
>  > UML the round trip times show considerable variabilty with gaps on the
>  >  order of seconds.
> 
> How's this for a theory:  The server's host is short on memory.  A UML 
> which
> is answering pings every second or so can be considered pretty idle, and 
> thus
> a good candidate to have its pages swapped out.  So, this happens, and when
> a ping comes in, UML needs to be swapped back in, causing your long and
> unpredictable delays.
> 
> Running a busy loop in the UML makes it busy from the point of view of the
> host and prevents it from being swapped out, giving you nice ping 
> latencies.
> 
>                                 Jeff
> 
> 





-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Q: UML thread communication - scheduling oddness

[Joe Marzot]

Jeff Dike wrote:
> gmarzot@nortelnetworks.com said:
>  > That is, if I run a little busy loop
>  >   /mnt/plankton/stress --cpu 1 --io 1
>  > on the server side, the round trip time for messages is greatly
>  > *improved*. without this activity and a basically dormant server side
>  > UML the round trip times show considerable variabilty with gaps on the
>  >  order of seconds.
> 
> How's this for a theory:  The server's host is short on memory.  A UML 
> which
> is answering pings every second or so can be considered pretty idle, and 
> thus
> a good candidate to have its pages swapped out.  So, this happens, and when
> a ping comes in, UML needs to be swapped back in, causing your long and
> unpredictable delays.

thinking about this a bit more...some doubts...note: the regular 'pings' 
which are serviced purely by kernel resident code/ kernel thread do not 
exhibit the delays. It is only when the communication involved a 
userspace app as the responder to the 'app-level-ping' that the delays 
come. Is the memory for the userspace app seperately swappable from that 
of the kernel thread? If so then the theory still makes sense, since the 
kernel thread likely has more going on so that is less likely to swap. 
If not, I am still confused.

-GSM

> 
> Running a busy loop in the UML makes it busy from the point of view of the
> host and prevents it from being swapped out, giving you nice ping 
> latencies.
> 
>                                 Jeff
> 
> 





-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Q: UML thread communication - scheduling oddness

[Jeff Dike]

gmarzot@nortelnetworks.com said:
> Could you add a few sentences though on the delivery mechnisms for the
>  the packets to the kernel thread and to userspace. I saw a poll() on
> the  pollfds array which has the taps fd in it but this is a
> non-blocking  poll and I could not trace to how often it is called and
> what triggers  it - I also saw the sigio handler but I could not see
> who was generating  the SIGIO - is that how the userspace thread is
> woken up? 

OK, an incoming packet is handled this way:
	it's received by the host and routed to the tap device (or 
alternatively, it's received by the host eth, sent to the bridge that it's part
of, and copied to the tap device which is also in the bridge)
	it pops out of /dev/net/tun, which UML has a file descriptor open to
	the file descriptor is set up O_ASYNC, so that generates a SIGIO
	this causes sigio_handler to be called
	it looks at all of the descriptors in its care, and handles all the
ones that are now active
	the network driver's interrupt routine is called whenever a tap fd is
active, and it reads the frame off the fd and passes it to the UML network 
stack

sigio_handler does blocking polls because it knows that something is active,
and that it won't, in fact, block.

				Jeff



-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Q: UML thread communication - scheduling oddness

[Jeff Dike]

gmarzot@nortelnetworks.com said:
> thinking about this a bit more...some doubts...note: the regular
> 'pings'  which are serviced purely by kernel resident code/ kernel
> thread do not  exhibit the delays. It is only when the communication
> involved a  userspace app as the responder to the 'app-level-ping'
> that the delays  come. Is the memory for the userspace app seperately
> swappable from that  of the kernel thread?

The userspace app is all of UML, including the parts of the kernel which
respond to pings.  It can all be swapped out by the host.

				Jeff



-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel