[uml-devel] mlock discussion is back - was: Re: [uml-user] odd ping problems

[uml-devel] mlock discussion is back - was: Re: [uml-user] odd ping problems

[roland]

hi!
since i had problems with those ping delays, too - i spent some time on that and i think 
i probably have found a relationship between uml ping delays and page-faults of the uml 
process.

if you don`t see the ping delays, you should be able to produce them with the following 
"receipe" - at least i`m able to reproduce it very easily this way:

get eatmem.c from http://www.theshore.net/~caker/patches/eatmem.c and compile.

now check your free memory (vmstat / top / free....) and let eatmem eat it up almost completely. 
(eatmem bigvalue loop)

after that, run "dd if=/dev/hda of=/dev/null" in a separate window.

you should see ping time (pinging from host to uml) go up from ~0.15ms to very much higher values.

watch your uml process with "sar" (install sysstat rpm - should be delivered with your distro - 
or get it from : http://perso.wanadoo.fr/sebastien.godard/)

sar -x umlpid1 -x umlpid2 -x umlpid3 -x umlpid4  1 0  (on skas host)

for at least two of the uml-pid`s you should see minor and major pagefaults (minflt/majflt) - i would 
say there is a releationship between those pagefaults and the ping delays. (i had turned off swap 
completely - so my host was not able to swap at all!)

can somebody acknowledge this ?

so - the question is: if those pagefaults cause that ping delays - how to stop them from happening
entirely ? paging is a quite common thing happening to processes - in every OS.

there was a controversial discussion about (optional) mlocking uml by a patch:
http://marc.theaimsgroup.com/?l=user-mode-linux-devel&m=107090239400338&w=2
at:
http://marc.theaimsgroup.com/?t=108150683900001&r=1&w=2

since we see, that this annoying effect happens over and over again and nobody can do something about
that - i`m not sure - but this patch probably could solve that obscure ping-delay/pagefault problem, 
too. so - why the hell not using and recommending it ?

reading into http://www.die.net/doc/linux/man/man2/mlockall.2.html:

>Memory locking has two main applications: real-time algorithms and high-security data processing. 
>Real-time applications require deterministic timing, and, like scheduling, !!!---> paging is one 
>major cause of unexpected program execution delays.<---!!! Real-time applications will usually also 
>switch to a real-time scheduler with sched_setscheduler. 

sorry if some people will roll their eyes now - but i`d like to bring michel pollet`s patch back into 
discussion again - because the ping (and probably other) delay effect(s) seem to be a result of paging 
(and not only swapping).  so, making sure that a host has enough RAM and making sure, that he won`t swap 
is definetly NOT enough to make sure that a uml runs smoothly.  some pagefaults on the uml process - pooof 
- and we have significant delays....

michel - if you are reading this - and if your mlock patch solves this problem - i`m sure some people would 
be happy if the mlock patch would be actively maintained separately and a port to 2.6.x would be done, too.
at least i`m very interested in a port to 2.6.x 
the discussion about this patch being merged or not can be made at a later time.....

regards
roland


----- Original Message ----- 
From: "Christopher S. Aker" <caker@theshore.net>
To: "roland" <for_spam@gmx.de>; "Johannes Berg" <johannes@sipsolutions.de>
Cc: <user-mode-linux-user@lists.sourceforge.net>
Sent: Wednesday, May 26, 2004 7:55 PM
Subject: Re: [uml-user] odd ping problems


> > hi !
> > maybe this is interesting for you:
> > http://marc.theaimsgroup.com/?t=108150683900001&r=1&w=2
> > http://marc.theaimsgroup.com/?t=108153567000002&r=1&w=2
> > ??
> 
> Thanks for the links.  This wasn't an ARP or I/O problem -- My testing was done
> from the host to a guest within the same subnet, avoiding the switch/networking
> gear all together.  The host had nothing in swap, the guest wasn't swapping or
> doing anything but responding to my pings along with an ssh session.
> 
> I'm pretty sure this is some weirdness in the host and/or UML kernel.  If I find
> the time, I'll try to reproduce.
> 
> Thanks!
> -Chris
> 


> johannes:~$ ping uml
> PING uml (172.17.16.100) 56(84) bytes of data.
> 64 bytes from uml (172.17.16.100): icmp_seq=2 ttl=64 time=10008 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=3 ttl=64 time=9008 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=4 ttl=64 time=8008 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=5 ttl=64 time=7009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=6 ttl=64 time=6009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=7 ttl=64 time=5009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=8 ttl=64 time=4008 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=9 ttl=64 time=3009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=10 ttl=64 time=2009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=11 ttl=64 time=1010 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=12 ttl=64 time=2.67 ms

Just a "me too" reply.  I noticed this tonight, while nmap'ing a guest.  Pings
start out very long and consistantly return back to normal response times.  Very
strange.  Using bridging/tuntap setup.

I haven't look farther into this.  Jeff, any thoughts?

-Chris



-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] mlock discussion is back - was: Re: [uml-user] odd ping problems

[roland]

hi!
i tried to forward-port the mlock patch to 2.6.6 - and succeeded so far, because it`s a small patch
and i just needed reading through the lines and adjust them to the right lines in  uml code.

unfortunately, it doesn`t seem to cure the ping-problem.
using sar,  i still see major pagefaults for the uml "master/root" process, while "hogging" the
host vm (eatmem, dd ....). btw: how can we distinguish between the different uml processes at runtime?

furthermore, i intuitively tried  mlockall(MCL_CURRENT| MCL_FUTURE) instead of mlock(addr,next_len) (see
http://www.die.net/doc/linux/man/man2/mlockall.2.html) and it seems this gives good results (no major
pagefaults anymore) - but i don`t really know what i`m doing here and if i`m doing it right  - because i`m
no kernel hacker :)
someone with more detailed knowledge could give a comment to this?

regards
roland


----- Original Message ----- 
From: "roland" <for_spam@gmx.de>
To: "Christopher S. Aker" <caker@theshore.net>; "Johannes Berg" <johannes@sipsolutions.de>
Cc: <user-mode-linux-user@lists.sourceforge.net>; <user-mode-linux-devel@lists.sourceforge.net>; <matthew-list@bytemark.co.uk>;
<michel@pollet.net>; "Jeff Dike" <jdike@addtoit.com>
Sent: Thursday, May 27, 2004 1:23 AM
Subject: [uml-devel] mlock discussion is back - was: Re: [uml-user] odd ping problems


> hi!
> since i had problems with those ping delays, too - i spent some time on that and i think
> i probably have found a relationship between uml ping delays and page-faults of the uml
> process.
>
> if you don`t see the ping delays, you should be able to produce them with the following
> "receipe" - at least i`m able to reproduce it very easily this way:
>
> get eatmem.c from http://www.theshore.net/~caker/patches/eatmem.c and compile.
>
> now check your free memory (vmstat / top / free....) and let eatmem eat it up almost completely.
> (eatmem bigvalue loop)
>
> after that, run "dd if=/dev/hda of=/dev/null" in a separate window.
>
> you should see ping time (pinging from host to uml) go up from ~0.15ms to very much higher values.
>
> watch your uml process with "sar" (install sysstat rpm - should be delivered with your distro -
> or get it from : http://perso.wanadoo.fr/sebastien.godard/)
>
> sar -x umlpid1 -x umlpid2 -x umlpid3 -x umlpid4  1 0  (on skas host)
>
> for at least two of the uml-pid`s you should see minor and major pagefaults (minflt/majflt) - i would
> say there is a releationship between those pagefaults and the ping delays. (i had turned off swap
> completely - so my host was not able to swap at all!)
>
> can somebody acknowledge this ?
>
> so - the question is: if those pagefaults cause that ping delays - how to stop them from happening
> entirely ? paging is a quite common thing happening to processes - in every OS.
>
> there was a controversial discussion about (optional) mlocking uml by a patch:
> http://marc.theaimsgroup.com/?l=user-mode-linux-devel&m=107090239400338&w=2
> at:
> http://marc.theaimsgroup.com/?t=108150683900001&r=1&w=2
>
> since we see, that this annoying effect happens over and over again and nobody can do something about
> that - i`m not sure - but this patch probably could solve that obscure ping-delay/pagefault problem,
> too. so - why the hell not using and recommending it ?
>
> reading into http://www.die.net/doc/linux/man/man2/mlockall.2.html:
>
> >Memory locking has two main applications: real-time algorithms and high-security data processing.
> >Real-time applications require deterministic timing, and, like scheduling, !!!---> paging is one
> >major cause of unexpected program execution delays.<---!!! Real-time applications will usually also
> >switch to a real-time scheduler with sched_setscheduler.
>
> sorry if some people will roll their eyes now - but i`d like to bring michel pollet`s patch back into
> discussion again - because the ping (and probably other) delay effect(s) seem to be a result of paging
> (and not only swapping).  so, making sure that a host has enough RAM and making sure, that he won`t swap
> is definetly NOT enough to make sure that a uml runs smoothly.  some pagefaults on the uml process - pooof
> - and we have significant delays....
>
> michel - if you are reading this - and if your mlock patch solves this problem - i`m sure some people would
> be happy if the mlock patch would be actively maintained separately and a port to 2.6.x would be done, too.
> at least i`m very interested in a port to 2.6.x
> the discussion about this patch being merged or not can be made at a later time.....
>
> regards
> roland
>
>
> ----- Original Message ----- 
> From: "Christopher S. Aker" <caker@theshore.net>
> To: "roland" <for_spam@gmx.de>; "Johannes Berg" <johannes@sipsolutions.de>
> Cc: <user-mode-linux-user@lists.sourceforge.net>
> Sent: Wednesday, May 26, 2004 7:55 PM
> Subject: Re: [uml-user] odd ping problems
>
>
> > > hi !
> > > maybe this is interesting for you:
> > > http://marc.theaimsgroup.com/?t=108150683900001&r=1&w=2
> > > http://marc.theaimsgroup.com/?t=108153567000002&r=1&w=2
> > > ??
> >
> > Thanks for the links.  This wasn't an ARP or I/O problem -- My testing was done
> > from the host to a guest within the same subnet, avoiding the switch/networking
> > gear all together.  The host had nothing in swap, the guest wasn't swapping or
> > doing anything but responding to my pings along with an ssh session.
> >
> > I'm pretty sure this is some weirdness in the host and/or UML kernel.  If I find
> > the time, I'll try to reproduce.
> >
> > Thanks!
> > -Chris
> >
>
>
> > johannes:~$ ping uml
> > PING uml (172.17.16.100) 56(84) bytes of data.
> > 64 bytes from uml (172.17.16.100): icmp_seq=2 ttl=64 time=10008 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=3 ttl=64 time=9008 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=4 ttl=64 time=8008 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=5 ttl=64 time=7009 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=6 ttl=64 time=6009 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=7 ttl=64 time=5009 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=8 ttl=64 time=4008 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=9 ttl=64 time=3009 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=10 ttl=64 time=2009 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=11 ttl=64 time=1010 ms
> > 64 bytes from uml (172.17.16.100): icmp_seq=12 ttl=64 time=2.67 ms
>
> Just a "me too" reply.  I noticed this tonight, while nmap'ing a guest.  Pings
> start out very long and consistantly return back to normal response times.  Very
> strange.  Using bridging/tuntap setup.
>
> I haven't look farther into this.  Jeff, any thoughts?
>
> -Chris
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle 10g.
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> _______________________________________________
> User-mode-linux-devel mailing list
> User-mode-linux-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
>



-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] mlock discussion is back - was: Re: [uml-user] odd ping problems

[Michel]

(sorry for the top-reply, it's pretty out of context anyway)

Hi Roland,

mlockall is not what you want, mlockall will lock all current pages, but also 
FUTURE pages *including memory mappings* so you definitly don't want to do 
that, thats totaly uncontrolable.
AFAIK the mlock patch works great (you remembered you needed to launch the UML 
kernel as root?) we've been using it on lots of UMLs at Bytemarks and it 
definitly totaly cured the problem of the long pings, and also makes all of 
the VM much, much more responsive in general.

What that patch solves is an issue that would be very, very difficult to fix 
without the uml-kernel having access to which host-kernel pages are in or out 
of swap.
The problem is that the uml-kernel assumes that his memory pages are... in 
memory so it has immediate access to them, so when it (uml-kernel) knows 
about a page fault for a particular task, it can just suspend *that task* 
until the page is available, and continue doing whatever it was doing 
(including responding to ping, and scheduling the other tasks) in the 
meantime.
In our case, the uml-kernel thinks it has immediate access to it's "memory" 
page, SAVE that the host-kernel has decided to page it. Therefore the 
uml-kernel gets locked by host-kernel, including all it's tasks. Thats how 
you end up with massive lockups on the UMLs, because everything gets stopped.

So that mlock patch prevents that by ensuring that the uml-kernel is always 
rigth when it thinks a memory page is... in memory. I agree it's a fairly 
large gun to kill a seemingly simple problem...

I really don't know how one would make the system work without giving a notion 
of the host-kernel page layout to the uml-kernel, and that I think would be a 
massive task to implement, as well as a potentialy fatal security risk.

For the patch itself, I was one of the early user having that particular issue 
at Bytemark, and therefore made a proof-of-concept patch to test, it worked 
so well that Matthew Bloch "cleaned" the patch, added the command line option 
etc. It's been running perfectly well on hundreds of VMs on production 
machines for months now.
So technicaly Matthew's more of the maintainer than I am :-)

Feel free to forward this to appropriate lists, I am not on them.

Thanks
Michel

On Thursday 27 May 2004 16:39, roland wrote:
> hi!
> i tried to forward-port the mlock patch to 2.6.6 - and succeeded so far,
> because it`s a small patch and i just needed reading through the lines and
> adjust them to the right lines in  uml code.
>
> unfortunately, it doesn`t seem to cure the ping-problem.
> using sar,  i still see major pagefaults for the uml "master/root" process,
> while "hogging" the host vm (eatmem, dd ....). btw: how can we distinguish
> between the different uml processes at runtime?
>
> furthermore, i intuitively tried  mlockall(MCL_CURRENT| MCL_FUTURE) instead
> of mlock(addr,next_len) (see
> http://www.die.net/doc/linux/man/man2/mlockall.2.html) and it seems this
> gives good results (no major pagefaults anymore) - but i don`t really know
> what i`m doing here and if i`m doing it right  - because i`m no kernel
> hacker :)
> someone with more detailed knowledge could give a comment to this?
>
> regards
> roland
>
>
> ----- Original Message -----
> From: "roland" <for_spam@gmx.de>
> To: "Christopher S. Aker" <caker@theshore.net>; "Johannes Berg"
> <johannes@sipsolutions.de> Cc:
> <user-mode-linux-user@lists.sourceforge.net>;
> <user-mode-linux-devel@lists.sourceforge.net>;
> <matthew-list@bytemark.co.uk>; <michel@pollet.net>; "Jeff Dike"
> <jdike@addtoit.com>
> Sent: Thursday, May 27, 2004 1:23 AM
> Subject: [uml-devel] mlock discussion is back - was: Re: [uml-user] odd
> ping problems
>
> > hi!
> > since i had problems with those ping delays, too - i spent some time on
> > that and i think i probably have found a relationship between uml ping
> > delays and page-faults of the uml process.
> >
> > if you don`t see the ping delays, you should be able to produce them with
> > the following "receipe" - at least i`m able to reproduce it very easily
> > this way:
> >
> > get eatmem.c from http://www.theshore.net/~caker/patches/eatmem.c and
> > compile.
> >
> > now check your free memory (vmstat / top / free....) and let eatmem eat
> > it up almost completely. (eatmem bigvalue loop)
> >
> > after that, run "dd if=/dev/hda of=/dev/null" in a separate window.
> >
> > you should see ping time (pinging from host to uml) go up from ~0.15ms to
> > very much higher values.
> >
> > watch your uml process with "sar" (install sysstat rpm - should be
> > delivered with your distro - or get it from :
> > http://perso.wanadoo.fr/sebastien.godard/)
> >
> > sar -x umlpid1 -x umlpid2 -x umlpid3 -x umlpid4  1 0  (on skas host)
> >
> > for at least two of the uml-pid`s you should see minor and major
> > pagefaults (minflt/majflt) - i would say there is a releationship between
> > those pagefaults and the ping delays. (i had turned off swap completely -
> > so my host was not able to swap at all!)
> >
> > can somebody acknowledge this ?
> >
> > so - the question is: if those pagefaults cause that ping delays - how to
> > stop them from happening entirely ? paging is a quite common thing
> > happening to processes - in every OS.
> >
> > there was a controversial discussion about (optional) mlocking uml by a
> > patch:
> > http://marc.theaimsgroup.com/?l=user-mode-linux-devel&m=107090239400338&w
> >=2 at:
> > http://marc.theaimsgroup.com/?t=108150683900001&r=1&w=2
> >
> > since we see, that this annoying effect happens over and over again and
> > nobody can do something about that - i`m not sure - but this patch
> > probably could solve that obscure ping-delay/pagefault problem, too. so -
> > why the hell not using and recommending it ?
> >
> > reading into http://www.die.net/doc/linux/man/man2/mlockall.2.html:
> > >Memory locking has two main applications: real-time algorithms and
> > > high-security data processing. Real-time applications require
> > > deterministic timing, and, like scheduling, !!!---> paging is one major
> > > cause of unexpected program execution delays.<---!!! Real-time
> > > applications will usually also switch to a real-time scheduler with
> > > sched_setscheduler.
> >
> > sorry if some people will roll their eyes now - but i`d like to bring
> > michel pollet`s patch back into discussion again - because the ping (and
> > probably other) delay effect(s) seem to be a result of paging (and not
> > only swapping).  so, making sure that a host has enough RAM and making
> > sure, that he won`t swap is definetly NOT enough to make sure that a uml
> > runs smoothly.  some pagefaults on the uml process - pooof - and we have
> > significant delays....
> >
> > michel - if you are reading this - and if your mlock patch solves this
> > problem - i`m sure some people would be happy if the mlock patch would be
> > actively maintained separately and a port to 2.6.x would be done, too. at
> > least i`m very interested in a port to 2.6.x
> > the discussion about this patch being merged or not can be made at a
> > later time.....
> >
> > regards
> > roland
> >
> >
> > ----- Original Message -----
> > From: "Christopher S. Aker" <caker@theshore.net>
> > To: "roland" <for_spam@gmx.de>; "Johannes Berg"
> > <johannes@sipsolutions.de> Cc:
> > <user-mode-linux-user@lists.sourceforge.net>
> > Sent: Wednesday, May 26, 2004 7:55 PM
> > Subject: Re: [uml-user] odd ping problems
> >
> > > > hi !
> > > > maybe this is interesting for you:
> > > > http://marc.theaimsgroup.com/?t=108150683900001&r=1&w=2
> > > > http://marc.theaimsgroup.com/?t=108153567000002&r=1&w=2
> > > > ??
> > >
> > > Thanks for the links.  This wasn't an ARP or I/O problem -- My testing
> > > was done from the host to a guest within the same subnet, avoiding the
> > > switch/networking gear all together.  The host had nothing in swap, the
> > > guest wasn't swapping or doing anything but responding to my pings
> > > along with an ssh session.
> > >
> > > I'm pretty sure this is some weirdness in the host and/or UML kernel. 
> > > If I find the time, I'll try to reproduce.
> > >
> > > Thanks!
> > > -Chris
> > >
> > >
> > >
> > > johannes:~$ ping uml
> > > PING uml (172.17.16.100) 56(84) bytes of data.
> > > 64 bytes from uml (172.17.16.100): icmp_seq=2 ttl=64 time=10008 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=3 ttl=64 time=9008 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=4 ttl=64 time=8008 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=5 ttl=64 time=7009 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=6 ttl=64 time=6009 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=7 ttl=64 time=5009 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=8 ttl=64 time=4008 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=9 ttl=64 time=3009 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=10 ttl=64 time=2009 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=11 ttl=64 time=1010 ms
> > > 64 bytes from uml (172.17.16.100): icmp_seq=12 ttl=64 time=2.67 ms
> >
> > Just a "me too" reply.  I noticed this tonight, while nmap'ing a guest. 
> > Pings start out very long and consistantly return back to normal response
> > times.  Very strange.  Using bridging/tuntap setup.
> >
> > I haven't look farther into this.  Jeff, any thoughts?
> >
> > -Chris
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: Oracle 10g
> > Get certified on the hottest thing ever to hit the market... Oracle 10g.
> > Take an Oracle 10g class now, and we'll give you the exam FREE.
> > http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> > _______________________________________________
> > User-mode-linux-devel mailing list
> > User-mode-linux-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

-- 
Linux yawn 2.6.5-mm6 #73 Sun May 9 18:19:52 BST 2004 i686


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

[uml-devel] Re: [uml-user] odd ping problems

[roland]

hi!
i`m able to reproduce this with 2.6.6-uml and tuntap/bridged network setup (on 
2.6.6 skas host).
this seems to have no relation to the swap/paging related ping delay issue - it 
seems to be a completely different one.
it only happens when "largepinging" from host to uml - not vice versa.

> The uml is idle during all the time, it reacts to my ssh session
> and the normal console. In fact, if I generate other network traffic
> (cat'ing a file in my ssh session), the uml will respond to the large
> ping.
your uml responds to the large ping? mine doesn`t respond to large pings
at all.

regards
roland


----- Original Message ----- 
From: "Johannes Berg" <johannes@sipsolutions.de>
To: <user-mode-linux-user@lists.sourceforge.net>
Sent: Monday, May 17, 2004 10:30 PM
Subject: [uml-user] odd ping problems


> Hi,
> 
> Just observed this on
>   host:  2.6.3 with skas
>   guest: debian uml package (2.4.24-1um)
> 
> johannes:~$ ping uml
> PING uml (172.17.16.100) 56(84) bytes of data.
> 64 bytes from uml (172.17.16.100): icmp_seq=1 ttl=64 time=1.00 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=2 ttl=64 time=0.538 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=3 ttl=64 time=0.519 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=4 ttl=64 time=0.530 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=5 ttl=64 time=0.529 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=6 ttl=64 time=1.92 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=7 ttl=64 time=0.499 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=8 ttl=64 time=6.64 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=9 ttl=64 time=1.36 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=10 ttl=64 time=0.519 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=11 ttl=64 time=0.528 ms
>  
> --- uml ping statistics ---
> 11 packets transmitted, 11 received, 0% packet loss, time 10027ms
> rtt min/avg/max/mdev = 0.499/1.327/6.642/1.738 ms
> 
> johannes:~$ ping uml -s 60000
> PING uml (172.17.16.100) 60000(60028) bytes of data.
>  
> --- uml ping statistics ---
> 5 packets transmitted, 0 received, 100% packet loss, time 20042ms
> 
> johannes:~$ ping uml
> PING uml (172.17.16.100) 56(84) bytes of data.
> 64 bytes from uml (172.17.16.100): icmp_seq=2 ttl=64 time=10008 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=3 ttl=64 time=9008 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=4 ttl=64 time=8008 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=5 ttl=64 time=7009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=6 ttl=64 time=6009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=7 ttl=64 time=5009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=8 ttl=64 time=4008 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=9 ttl=64 time=3009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=10 ttl=64 time=2009 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=11 ttl=64 time=1010 ms
> 64 bytes from uml (172.17.16.100): icmp_seq=12 ttl=64 time=2.67 ms
>  
> --- uml ping statistics ---
> 12 packets transmitted, 11 received, 8% packet loss, time 11007ms
> rtt min/avg/max/mdev = 2.676/5008.647/10008.979/3162.981 ms, pipe 11
> 
> After this it'll react fine to my pings, until again I send it a very
> large one. That blocks it for a while again, and I get behaviour like
> above. Notice the large delay, until all the sudden all packets arrive
> (ping sends one every second, but the first 12 arrive at the same time).
> Sometimes, instead of all arriving late, the packets are simply dropped.
> 
> The uml is idle during all the time, it reacts to my ssh session
> and the normal console. In fact, if I generate other network traffic
> (cat'ing a file in my ssh session), the uml will respond to the large
> ping.
> 
> Is this a known problem?
> 
> More information:
> The uml has iptables enabled, and drops every icmp packet but "ping pong
> destination-unreachable time-exceeded", but I can reproduce exactly the
> same behaviour if I flush all chains and set all policies to "accept".
> 
> I am using
>     eth0=daemon,,,/var/run/uml-utilities/uml_switch.ctl
> 
> The strace looks similar to what I had before, when I suspended my laptop
> while uml was running.
> 
> Basically, during the pinging with a small packet, the uml will do:
> recvfrom(7, "...", 1514, 0, NULL, NULL) = 98
> <rt_sigprocmask unblock + block>
> gettimeofday
> <rt_sigprocmask unblock + block * 9>
> recvfrom(7, "...", 1514, 0, NULL, NULL) = -1 (EAGAIN)
> <rt_sigprocmask unblock + block * 6>
> ioctl(7, SNDCTL_TMR_TIMEBASE or TCGETS, 0xa021b3a4) = -1 EINVAL
> <rt_sigprocmask unblock + block * 15>
> sendto(7, "...", 98, 0, {sa_family=AF_UNIX, path=@}, 110) = 98
> 
> From the first recvfrom to the sendto it takes about 5673 usecs.
> 
> For the fragmented packets, the recvfrom sequence (lots of receives of
> course) takes 15138 usecs. But before the next sendto it takes more than 3
> seconds, there are other recvfroms inbetween, and some waitpid stuff, and
> ptrace, ... It also sleeps for a while, sequences like
> 22:17:31.647087 nanosleep({10, 0}, 0)   = ? ERESTART_RESTARTBLOCK (To be restarted)
> 22:17:31.657382 --- SIGALRM (Alarm clock) @ 0 (0) ---
> 
> Thats about all I can say here, I got the data with
>   strace -tt -p <pid>
> 
> johannes
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: SourceForge.net Broadband
> Sign-up now for SourceForge Broadband and get the fastest
> 6.0/768 connection for only $19.95/mo for the first 3 months!
> http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
> _______________________________________________
> User-mode-linux-user mailing list
> User-mode-linux-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/user-mode-linux-user
> 


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

[uml-devel] Re: [uml-user] odd ping problems

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 226 bytes --]

On Thu, 2004-05-27 at 18:39, roland wrote:
> your uml responds to the large ping? mine doesn`t respond to large pings
> at all.

Yes, it definitely responds to a large ping, see original transcript I
posted.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [uml-devel] mlock discussion is back - was: Re: [uml-user] odd ping problems

[BlaisorBlade]

Alle 01:23, giovedì 27 maggio 2004, roland ha scritto:
> for at least two of the uml-pid`s you should see minor and major pagefaults
> (minflt/majflt) - i would say there is a releationship between those
> pagefaults and the ping delays. (i had turned off swap completely - so my
> host was not able to swap at all!)

That is not very good - if the latency problem comes from UML being swapped 
out, why do you disable swapping? Actually there is some "swapping", in a 
certain sense, so you could observe the major page faults (major means 
"requiring I/O", minor means "no IO": this can happen for COW pages, to mark 
as dirty a clean page).

When swapping is disabled, the pages belonging to a memory mapping, that was 
not modified since they were read from the disk, can be swapped out.

I.e. the executable pages, actually, are created by mmap'ing the executable; 
since they are not modified, they can be removed from memory.

Also the data pages which come from the executable can be treated the same 
way, if they are still "clean" (i.e. not modified).

[ I actually checked the code works this way, i.e. if you try to read a global 
var, even if it's mmaped with PROT_WRITE, in the page-tables it is marked as 
read-only; when you try to write it, it will be marked as read-write and the 
page will become dirty. ]

-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729



-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
From Windows to Linux, servers to mobile, InstallShield X is the one
installation-authoring solution that does it all. Learn more and
evaluate today! http://www.installshield.com/Dev2Dev/0504
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] mlock discussion is back - was: Re: [uml-user] odd ping problems

[Henrik Nordstrom]

On Wed, 2 Jun 2004, BlaisorBlade wrote:

> That is not very good - if the latency problem comes from UML being swapped 
> out, why do you disable swapping? Actually there is some "swapping", in a 
> certain sense, so you could observe the major page faults (major means 
> "requiring I/O", minor means "no IO": this can happen for COW pages, to mark 
> as dirty a clean page).
> 
> When swapping is disabled, the pages belonging to a memory mapping, that was 
> not modified since they were read from the disk, can be swapped out.

In case of UML the situation is even worse unless /dev/anon is used. As
UML then makes the kernel memory by mmap:ing some temporary files if
effectively makes a new swap space to where the UMLs will be swapped,
making UMLs or UML memory very likely candidates to go out of memory
should there be some memory pressure on the host.  Disabling the swap 
space only makes it less likely other non-UML applications gets swapped 
out.

If you run with the temporary files on tmpfs then this pageing of the UML 
data can be disabled I think, but not verified.

Regards
Henrik




-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
From Windows to Linux, servers to mobile, InstallShield X is the one
installation-authoring solution that does it all. Learn more and
evaluate today! http://www.installshield.com/Dev2Dev/0504
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel