* runt Tactful Small Carton, International Courier. alleviate
From: fritz floren @ 2005-09-26 19:16 UTC (permalink / raw)
To: Clifton Ashworth
Cc: glover, terry, mendoza, lowe, netdev, daniel, linux-xfs-outgoing
Set a snug e zonne setting.
Together we eliminate malady on diarrhoea.
Straightforward to tracks your delivery with our system.
One big sells. Sale to all remedial.
Online physician can consult for zero pennies.
I am very suggested purchasing from this dealer. Kath J --NJ.
http://uk.geocities.com/wonderfulcomplimentwell/?atednkky
I gathered from the conversation that Mr. Skimpole had been educated for
the medical profession and had once lived, in his professional capacity, in
the household of a German prince. He told us, however, that as he had always
been a mere child in point of weights and measures and had never known
anything about them (except that they disgusted him), he had never been able
to prescribe with the requisite accuracy of detail. In fact, he said, he had
no head for detail. And he told us, with great humour, that when he was
wanted to bleed the prince or physic any of his people, he was generally
found lying on his back in bed, reading the newspapers or making
fancy-sketches in pencil, and couldn't come. The prince, at last, objecting
to this, "in which," said Mr. Skimpole, in the frankest manner, "he was
perfectly right," the engagement terminated, and Mr. Skimpole having (as he
added with delightful gaiety) "nothing to live upon but love, fell in love,
and married, and surrounded himself with rosy cheeks." His good friend
Jarndyce and some other of his good friends then helped him, in quicker or
slower succession, to several openings in life, but to no purpose, for he
must confess to two of the oldest infirmities in the world: one was that he
had no idea of time, the other that he had no idea of money. In consequence
of which he never kept an appointment, never could transact any business,
and never knew the value of anything. Well. So he had got on in life, and
here he was. He was very fond of reading the papers, very fond of making
fancy-sketches with a pencil, very fond of nature, very fond of art. All he
asked of society was to let him live. THAT wasn't much. His wants were few.
Give him the papers, conversation, music, mutton, coffee, landscape, fruit
in the season, a few sheets of Bristol-board, and a little claret, and he
asked no more. He was a mere child in the world, but he didn't cry for the
moon. He said to the world, "Go your several ways in peace. Wear red coats,
blue coats, lawn sleeves; put pens behind your ears, wear aprons; go after
glory, holiness, commerce, trade, any object you prefer; only--let Harold
Skimpole live."
efficientgnicer emxs flchargefee AH02 filinge filejet
"Who?"
"Oh, it doesn't matter much. One of them. Only not Mac. I'm too fond of
him."
^ permalink raw reply
* Re: ipvs_syncmaster brings cpu to 100%
From: Nishanth Aravamudan @ 2005-09-26 17:51 UTC (permalink / raw)
To: Luca Maranzano; +Cc: LinuxVirtualServer.org users mailing list., netdev
In-Reply-To: <68559cef05092607441dd8e961@mail.gmail.com>
On 26.09.2005 [16:44:09 +0200], Luca Maranzano wrote:
> On 26/09/05, Nishanth Aravamudan <nacc@us.ibm.com> wrote:
> > On 26.09.2005 [15:52:02 +0200], Luca Maranzano wrote:
> > > On 26/09/05, Nishanth Aravamudan <nacc@us.ibm.com> wrote:
> > > > On 26.09.2005 [17:12:32 +0900], Horms wrote:
> > > > > On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:
> > > > >
> > > > > [snip]
> > > > >
> > > > > > > > > > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > > > > > > > > > the function schedule_timeout() is more used than the ssleep() (517
> > > > > > > > > > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> > > > > > > > > >
> > > > > > > > > > The other oddity is that Horms reported on this list that on non Xeon
> > > > > > > > > > CPU the same version of kernel of mine does not present the problem.
> > > > > > > > > >
> > > > > > > > > > I'm getting crazy :-)
> > > > > > > >
> > > > > > > > I've prepared a patch, which reverts the change which was introduced
> > > > > > > > by Nishanth Aravamudan in February.
> > > > > > >
> > > > > > > Was the 100% cpu utilization only occurring on Xeon processors?
> > > > > >
> > > > > > That seems to be the only case where were this problem has been
> > > > > > observed. I don't have such a processor myself, so I haven't actually
> > > > > > been able to produce the problem locally.
> > > > > >
> > > > > > One reason I posted this issue to netdev was to get some more
> > > > > > eyes on the problem as it is puzzling to say the least.
> > > > > >
> > > > > > > Care to try to use msleep_interruptible() instead of ssleep(), as
> > > > > > > opposed to schedule_timeout()?
> > > > > >
> > > > > > I will send a version that does that shortly, Luca, can
> > > > > > you plase check that too?
> > > > >
> > > > > Here is that version of the patch. Nishanth, I take it that I do not
> > > > > need to set TASK_INTERRUPTABLE before calling msleep_interruptible(),
> > > > > please let me know if I am wrong.
> > > >
> > > > Yes, exactly. I'm just trying to narrow it down to see if it's the task
> > > > state that's causing the issue (which, to be honest, doesn't make a lot
> > > > of sense to me -- with ssleep() your load average will go up as the task
> > > > will be UNINTERRUPTIBLE state, but I am not sure why utilisation would
> > > > rise, as you are still sleeping...)
> >
> > [trimmed lvs-users from my reply, as it is a closed list]
> >
> > > Just to add more info, please note the output of "ps":
> > >
> > > debld1:~# ps aux|grep ipvs
> > > root 3748 0.0 0.0 0 0 ? D 12:09 0:00
> > > [ipvs_syncmaster]
> > > root 3757 0.0 0.0 0 0 ? D 12:09 0:00
> > > [ipvs_syncbackup]
> > >
> > > Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep
> > > (usually IO)
> >
> > The msleep_interruptible() change should fix that.
> >
> > But that does not show 100% CPU utilisation at all, it shows 0. Did you
> > mean to say your load increases?
> >
> > I'm still unclear what the problem is. Horms initial Cc trimmed some
> > important information. It would be very useful to "start over" -- at
> > least from the perspective of what the problem actually is.
> >
> > > I hope to have a Xeon machine to make some more tests in the next
> > > days, in the mean time I'll try to reproduce my setup on a couple of
> > > VMWare Workstation machines.
> >
> > Please don't top-most. It makes it really hard to write sane replies...
>
> [trimmed Cc to avoid spamming...]
>
> Ok, just to summarize the long thread from the beginning:
>
> The goal: setting up a Local Director with IPVS with state
> synchronization, failover and failback.
>
> The hardware: 1 CPU Intel Xeon 3,4 Ghz - HP DL380G4 on 2 identical boxes
>
> The problems (please note that all kernel versions are *Debian* kernels):
> 1. Kernel 2.6.8: got a system lock of the standby node when simulating
> a failover. The load average as reported from "top" or "w" is always
> 0.00.
>
> 2. Kernel 2.6.11 and Kernel 2.6.12: failover and failback works fine,
> but the load average as reported from "top" or "w" is always
> systematically at 2.00 or more with both sync thread started
> (ipvs_syncmaster and ipvs_syncbackup). Load average from top is 1.00
> or mroe with only one thread (i.e. ipvs_syncmaster). Horms reported
> that he was not able to reproduce this on a non-Xeon system.
Ok, so when whomever mentioned "CPU utilisation" they were mistaken. The
load average being 2 is due to ssleep(). The msleep_interruptible()
version of the patch should fix that up. It really doesn't make any
difference in the code, except that your load average will go back to
0.00 and the ipvs threads can be interrupted by signals.
I would expect the load average to be 2.00 for all systems, not just
Xeon. The system lock has nothing to do with the patch, though.
Something else fixed it.
Thanks,
Nish
P.S. Again, please don't top-post, it makes it harder for me to reply
(and disinclines me to do so).
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: Emmanuel Fleury @ 2005-09-26 16:31 UTC (permalink / raw)
To: Michael Bellion; +Cc: linux-kernel, netdev
In-Reply-To: <200509261803.28150.mbellion@hipac.org>
Michael Bellion wrote:
>
> Sorry, but this is far away from the worst case for your scheme. Actually it
> is a quite good case for your compiler, because every rule is fully specified
> (meaning there are no wildcards in any rule) and there are no ranges or masks
> involved.
> Try using a mixed rule set that contains rules that only specify certain
> dimensions and have wildcards on the other dimensions. Try using rules with
> ranges and masks.
> Try using overlapping rules, meaning rules that completely or partly overlap
> other rules in certain dimensions.
> This will make your data structure grow!
I think you misunderstood our experiment. In fact, we were trying to
generate as much possible different singletons on the domain (each of
our rule was the header of a packet which have not been seen before),
because if we can group these rules into intervals, then our scheme is
having some advantages.
We were using IDD (Interval Decision Diagrams) which is a kind of
extended BDD (Binary Decision Diagrams) where you take your decision by
looking at a partition of the possible values of the variable. For
example, looking at the value x in [0,1024] where [0,128] leads to one
node in the decision tree, [129,256] to another and [257,1024] to a last
one. More this partition is fragmented more you increase the size of the
structure. Having a lot of overlap does certainly increase the number of
partitions, but adding singletons is the simplest way to increase the
number of partitions.
Take a look at this paper, maybe you can get some idea for your scheme
(it might be that some hybrid between your ideas and ours can make it):
http://www.cs.aau.dk/~fleury/download/papers/tc04.pdf
> Yes, you are right. The HiPAC project has gone through some tough times over
> the last 2 years. With MARA Systems the HiPAC Project has finally found a
> strong partner that is fully committed to the concept of Open Source
> Software. This allows me to continue the development of HiPAC under the GNU
> GPL license.
I'm always happy to see a firm funding some Open Source project. So, I
can do anything else but wishing you good luck for the future. :)
> Ok, I'll do that :)
Good. :)
Regards
--
Emmanuel Fleury
As usual, goodness hardly puts up a fight.
-- Calvin & Hobbes (Bill Waterson)
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: Michael Bellion @ 2005-09-26 16:03 UTC (permalink / raw)
To: Emmanuel Fleury; +Cc: linux-kernel, netdev
In-Reply-To: <43380E4A.1060604@cs.aau.dk>
Hi,
> > But your performance tests have a serious flaw:
> > You construct your rule set by creating one rule for each entry in your
> > packet header trace. This results in an completely artificial rule set
> > that creates a lot of redundancy in the nf-HiPAC lookup data structure
> > making it much larger than the Compact Filter data structure.
>
> Yes, it was intended to be a worst case for our scheme (not realistic
> but worst case)..
Sorry, but this is far away from the worst case for your scheme. Actually it
is a quite good case for your compiler, because every rule is fully specified
(meaning there are no wildcards in any rule) and there are no ranges or masks
involved.
Try using a mixed rule set that contains rules that only specify certain
dimensions and have wildcards on the other dimensions. Try using rules with
ranges and masks.
Try using overlapping rules, meaning rules that completely or partly overlap
other rules in certain dimensions.
This will make your data structure grow!
> > I am currently working on a new improved version of the algorithm used in
> > nf-HiPAC. The new algorithmic core will reduce memory usage while at the
> > same time improving the running time of insert and delete operations. The
> > lookup performance will be improved too, especially for bigger rulesets.
> > The concepts and the design are already developed, but the implementation
> > is still in its early stages.
> >
> > The new algorithmic core will make sure that the lookup data structure in
> > the kernel is always fully optimized while at the same time allowing very
> > fast dynamic updates.
> >
> > At that point Compact Filter will not be able to win in any performance
> > test against nf-HiPAC anymore, simply because there is no way to
> > optimize the lookup data structure any further.
>
> Well, you already said this last time we had exchanged some mails
> (it was more than one year ago if I count well).
Yes, you are right. The HiPAC project has gone through some tough times over
the last 2 years. With MARA Systems the HiPAC Project has finally found a
strong partner that is fully committed to the concept of Open Source
Software. This allows me to continue the development of HiPAC under the GNU
GPL license.
> Anyway, I doubt you can get something that you can update dynamically
> AND small in size following your way of doing. But, prove me wrong and
> I'll be happy. :)
Ok, I'll do that :)
Regards,
+---------------------------+
| Michael Bellion |
| <mbellion@hipac.org> |
+---------------------------+
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: Emmanuel Fleury @ 2005-09-26 15:05 UTC (permalink / raw)
To: Michael Bellion, linux-kernel, netdev
In-Reply-To: <200509261638.12731.mbellion@hipac.org>
Michael Bellion wrote:
>
> The current version of the algorithm used in nf-HiPAC does not optimize
> certain aspects of the lookup data structure in order to increase the speed
> of dynamic rule set updates.
> This means that the lookup data structure is larger than it really needs to be
> because it contains some unnecessary redundancy.
Could you quantify how much this "unnecessary redundancy" does hit the
size of the filter. Because last time I looked it was quite huge (you
may have improve it). And having a fat kernel does not help in backbones.
> But your performance tests have a serious flaw:
> You construct your rule set by creating one rule for each entry in your packet
> header trace. This results in an completely artificial rule set that creates
> a lot of redundancy in the nf-HiPAC lookup data structure making it much
> larger than the Compact Filter data structure.
Yes, it was intended to be a worst case for our scheme (not realistic
but worst case). We were more interested in comparing the complexity of
the different algorithms better than the efficiency of several
implementations.
I don't consider this as a flaw in our experiment because our goal was
different from having a real proof of concept (kind of having an
empirical evidence of a theoretical result).
> You have to understand that with real world rule sets the size of the computed
> lookup data structure will not be much different for Compact Filter and
> nf-HiPAC. This means that when you use real world rule sets there shouldn't
> be any noticeable difference in lookup performance betweeen Compact Filter
> and nf-HiPAC.
Might be right, but admit that the big problem of your algorithm is the
size of your data-structure in kernel-space. What you gain in speed, you
loose it in memory. And this IS an issue on routers (IMHO).
> I am currently working on a new improved version of the algorithm used in
> nf-HiPAC. The new algorithmic core will reduce memory usage while at the same
> time improving the running time of insert and delete operations. The lookup
> performance will be improved too, especially for bigger rulesets. The
> concepts and the design are already developed, but the implementation is
> still in its early stages.
>
> The new algorithmic core will make sure that the lookup data structure in the
> kernel is always fully optimized while at the same time allowing very fast
> dynamic updates.
>
> At that point Compact Filter will not be able to win in any performance test
> against nf-HiPAC anymore, simply because there is no way to optimize the
> lookup data structure any further.
Well, you already said this last time we had exchanged some mails
(it was more than one year ago if I count well).
Anyway, I doubt you can get something that you can update dynamically
AND small in size following your way of doing. But, prove me wrong and
I'll be happy. :)
Regards
--
Emmanuel Fleury
Ideals are dangerous things. Realities are better.
They wound but they are better.
-- Oscar Wilde
^ permalink raw reply
* Re: ipvs_syncmaster brings cpu to 100%
From: Luca Maranzano @ 2005-09-26 14:44 UTC (permalink / raw)
To: Nishanth Aravamudan, LinuxVirtualServer.org users mailing list.; +Cc: netdev
In-Reply-To: <20050926142109.GD7532@us.ibm.com>
[trimmed Cc to avoid spamming...]
Ok, just to summarize the long thread from the beginning:
The goal: setting up a Local Director with IPVS with state
synchronization, failover and failback.
The hardware: 1 CPU Intel Xeon 3,4 Ghz - HP DL380G4 on 2 identical boxes
The problems (please note that all kernel versions are *Debian* kernels):
1. Kernel 2.6.8: got a system lock of the standby node when simulating
a failover. The load average as reported from "top" or "w" is always
0.00.
2. Kernel 2.6.11 and Kernel 2.6.12: failover and failback works fine,
but the load average as reported from "top" or "w" is always
systematically at 2.00 or more with both sync thread started
(ipvs_syncmaster and ipvs_syncbackup). Load average from top is 1.00
or mroe with only one thread (i.e. ipvs_syncmaster). Horms reported
that he was not able to reproduce this on a non-Xeon system.
That's all, let me know if you need more info.
Regards,
Luca
On 26/09/05, Nishanth Aravamudan <nacc@us.ibm.com> wrote:
> On 26.09.2005 [15:52:02 +0200], Luca Maranzano wrote:
> > On 26/09/05, Nishanth Aravamudan <nacc@us.ibm.com> wrote:
> > > On 26.09.2005 [17:12:32 +0900], Horms wrote:
> > > > On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:
> > > >
> > > > [snip]
> > > >
> > > > > > > > > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > > > > > > > > the function schedule_timeout() is more used than the ssleep() (517
> > > > > > > > > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> > > > > > > > >
> > > > > > > > > The other oddity is that Horms reported on this list that on non Xeon
> > > > > > > > > CPU the same version of kernel of mine does not present the problem.
> > > > > > > > >
> > > > > > > > > I'm getting crazy :-)
> > > > > > >
> > > > > > > I've prepared a patch, which reverts the change which was introduced
> > > > > > > by Nishanth Aravamudan in February.
> > > > > >
> > > > > > Was the 100% cpu utilization only occurring on Xeon processors?
> > > > >
> > > > > That seems to be the only case where were this problem has been
> > > > > observed. I don't have such a processor myself, so I haven't actually
> > > > > been able to produce the problem locally.
> > > > >
> > > > > One reason I posted this issue to netdev was to get some more
> > > > > eyes on the problem as it is puzzling to say the least.
> > > > >
> > > > > > Care to try to use msleep_interruptible() instead of ssleep(), as
> > > > > > opposed to schedule_timeout()?
> > > > >
> > > > > I will send a version that does that shortly, Luca, can
> > > > > you plase check that too?
> > > >
> > > > Here is that version of the patch. Nishanth, I take it that I do not
> > > > need to set TASK_INTERRUPTABLE before calling msleep_interruptible(),
> > > > please let me know if I am wrong.
> > >
> > > Yes, exactly. I'm just trying to narrow it down to see if it's the task
> > > state that's causing the issue (which, to be honest, doesn't make a lot
> > > of sense to me -- with ssleep() your load average will go up as the task
> > > will be UNINTERRUPTIBLE state, but I am not sure why utilisation would
> > > rise, as you are still sleeping...)
>
> [trimmed lvs-users from my reply, as it is a closed list]
>
> > Just to add more info, please note the output of "ps":
> >
> > debld1:~# ps aux|grep ipvs
> > root 3748 0.0 0.0 0 0 ? D 12:09 0:00
> > [ipvs_syncmaster]
> > root 3757 0.0 0.0 0 0 ? D 12:09 0:00
> > [ipvs_syncbackup]
> >
> > Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep
> > (usually IO)
>
> The msleep_interruptible() change should fix that.
>
> But that does not show 100% CPU utilisation at all, it shows 0. Did you
> mean to say your load increases?
>
> I'm still unclear what the problem is. Horms initial Cc trimmed some
> important information. It would be very useful to "start over" -- at
> least from the perspective of what the problem actually is.
>
> > I hope to have a Xeon machine to make some more tests in the next
> > days, in the mean time I'll try to reproduce my setup on a couple of
> > VMWare Workstation machines.
>
> Please don't top-most. It makes it really hard to write sane replies...
>
> Thanks,
> Nish
>
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: Michael Bellion @ 2005-09-26 14:38 UTC (permalink / raw)
To: Emmanuel Fleury; +Cc: linux-kernel, linux-net, netdev, jamal
In-Reply-To: <4337DA7C.2000804@cs.aau.dk>
Hi
> I'm still not convinced by your approach. :-/
You really should have a closer look at nf-HiPAC so that you know what you are
talking about!
Your Compact Filter takes a completely different approach than nf-HiPAC to
build the data structure used in the kernel for the packet classification
lookup.
Your Compact Filter uses a static compiler in user space. That compiler
transforms the rule set into boolean expressions and than uses operations
from predicate logic to optimize the rule set.
This has the big drawback that whenever only a single rule changes you have to
recompile the complete lookup data structure. So this approach is clearly not
suitable for scenarios depending on dynamic rule sets.
nf-HiPAC uses a completely different approach to build the lookup data
structure in the kernel. It is based on geometry.
This approach allows completely dynamic updates. During an update of the rules
only the required changes of the lookup data structure are made. The data
structure is NOT rebuild from scratch. This guarantees that the packet
processing is only affected to the least possible amount during updates.
Although nf-HiPAC and Compact Filter use completely different approaches and
algorithms to build the lookup data structure it is important that you
understand the following:
nf-HiPAC and Compact filter end up with a very very similar lookup data
structure in the kernel.
> These experiments have to be updated but can you comment on this:
> http://www.cs.aau.dk/~mixxel/cf/experiments.html
The current version of the algorithm used in nf-HiPAC does not optimize
certain aspects of the lookup data structure in order to increase the speed
of dynamic rule set updates.
This means that the lookup data structure is larger than it really needs to be
because it contains some unnecessary redundancy.
This explains your test results.
Compact Filter and nf-HiPAC perform the same when they are both able to keep
their lookup data structure in the CPU caches and when they are both not able
to do so anymore.
Compact Filter is currently able to perform better in the area where it is
able to keep its data structure still in the caches while nf-HiPAC is not
able to do so anymore.
Most aspects of your performance tests are quite nice (e.g. the generating the
traffic by replaying a packet header trace).
But your performance tests have a serious flaw:
You construct your rule set by creating one rule for each entry in your packet
header trace. This results in an completely artificial rule set that creates
a lot of redundancy in the nf-HiPAC lookup data structure making it much
larger than the Compact Filter data structure.
You have to understand that with real world rule sets the size of the computed
lookup data structure will not be much different for Compact Filter and
nf-HiPAC. This means that when you use real world rule sets there shouldn't
be any noticeable difference in lookup performance betweeen Compact Filter
and nf-HiPAC.
-----------------
I am currently working on a new improved version of the algorithm used in
nf-HiPAC. The new algorithmic core will reduce memory usage while at the same
time improving the running time of insert and delete operations. The lookup
performance will be improved too, especially for bigger rulesets. The
concepts and the design are already developed, but the implementation is
still in its early stages.
The new algorithmic core will make sure that the lookup data structure in the
kernel is always fully optimized while at the same time allowing very fast
dynamic updates.
At that point Compact Filter will not be able to win in any performance test
against nf-HiPAC anymore, simply because there is no way to optimize the
lookup data structure any further.
Regards,
+---------------------------+
| Michael Bellion |
| <mbellion@hipac.org> |
+---------------------------+
^ permalink raw reply
* Out of Office AutoReply: [FMG-SPAM] - AL J RAJITDQTQH - Bayesian Filter detected spam
From: Liesbet Nyssen @ 2005-09-26 14:30 UTC (permalink / raw)
To: netdev
I'm out of office until October 25th and will have only limited access to my email. Please, contact me again later. Liesbet Nyssen
^ permalink raw reply
* RE: [FMG-SPAM] - AL J RAJITDQTQH - Bayesian Filter detected spam
From: Liesbet Nyssen @ 2005-09-26 14:30 UTC (permalink / raw)
To: netdev
I'm out of office until October 25th and will have only limited access to my email. Please contact me again later. Liesbet Nyssen
^ permalink raw reply
* Re: ipvs_syncmaster brings cpu to 100%
From: Nishanth Aravamudan @ 2005-09-26 14:21 UTC (permalink / raw)
To: Luca Maranzano; +Cc: Dave Miller, Wensong Zhang, Julian Anastasov, netdev
In-Reply-To: <68559cef05092606521cc13f9a@mail.gmail.com>
On 26.09.2005 [15:52:02 +0200], Luca Maranzano wrote:
> On 26/09/05, Nishanth Aravamudan <nacc@us.ibm.com> wrote:
> > On 26.09.2005 [17:12:32 +0900], Horms wrote:
> > > On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:
> > >
> > > [snip]
> > >
> > > > > > > > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > > > > > > > the function schedule_timeout() is more used than the ssleep() (517
> > > > > > > > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> > > > > > > >
> > > > > > > > The other oddity is that Horms reported on this list that on non Xeon
> > > > > > > > CPU the same version of kernel of mine does not present the problem.
> > > > > > > >
> > > > > > > > I'm getting crazy :-)
> > > > > >
> > > > > > I've prepared a patch, which reverts the change which was introduced
> > > > > > by Nishanth Aravamudan in February.
> > > > >
> > > > > Was the 100% cpu utilization only occurring on Xeon processors?
> > > >
> > > > That seems to be the only case where were this problem has been
> > > > observed. I don't have such a processor myself, so I haven't actually
> > > > been able to produce the problem locally.
> > > >
> > > > One reason I posted this issue to netdev was to get some more
> > > > eyes on the problem as it is puzzling to say the least.
> > > >
> > > > > Care to try to use msleep_interruptible() instead of ssleep(), as
> > > > > opposed to schedule_timeout()?
> > > >
> > > > I will send a version that does that shortly, Luca, can
> > > > you plase check that too?
> > >
> > > Here is that version of the patch. Nishanth, I take it that I do not
> > > need to set TASK_INTERRUPTABLE before calling msleep_interruptible(),
> > > please let me know if I am wrong.
> >
> > Yes, exactly. I'm just trying to narrow it down to see if it's the task
> > state that's causing the issue (which, to be honest, doesn't make a lot
> > of sense to me -- with ssleep() your load average will go up as the task
> > will be UNINTERRUPTIBLE state, but I am not sure why utilisation would
> > rise, as you are still sleeping...)
[trimmed lvs-users from my reply, as it is a closed list]
> Just to add more info, please note the output of "ps":
>
> debld1:~# ps aux|grep ipvs
> root 3748 0.0 0.0 0 0 ? D 12:09 0:00
> [ipvs_syncmaster]
> root 3757 0.0 0.0 0 0 ? D 12:09 0:00
> [ipvs_syncbackup]
>
> Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep
> (usually IO)
The msleep_interruptible() change should fix that.
But that does not show 100% CPU utilisation at all, it shows 0. Did you
mean to say your load increases?
I'm still unclear what the problem is. Horms initial Cc trimmed some
important information. It would be very useful to "start over" -- at
least from the perspective of what the problem actually is.
> I hope to have a Xeon machine to make some more tests in the next
> days, in the mean time I'll try to reproduce my setup on a couple of
> VMWare Workstation machines.
Please don't top-most. It makes it really hard to write sane replies...
Thanks,
Nish
^ permalink raw reply
* Re: ipvs_syncmaster brings cpu to 100%
From: Luca Maranzano @ 2005-09-26 13:52 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: LinuxVirtualServer.org users mailing list., Dave Miller,
Wensong Zhang, Julian Anastasov, netdev
In-Reply-To: <20050926131104.GA7532@us.ibm.com>
Just to add more info, please note the output of "ps":
debld1:~# ps aux|grep ipvs
root 3748 0.0 0.0 0 0 ? D 12:09 0:00
[ipvs_syncmaster]
root 3757 0.0 0.0 0 0 ? D 12:09 0:00
[ipvs_syncbackup]
Note the D status, i.e. (from ps(1) man page): Uninterruptible sleep
(usually IO)
I hope to have a Xeon machine to make some more tests in the next
days, in the mean time I'll try to reproduce my setup on a couple of
VMWare Workstation machines.
More later.
Thank you all.
Luca
On 26/09/05, Nishanth Aravamudan <nacc@us.ibm.com> wrote:
> On 26.09.2005 [17:12:32 +0900], Horms wrote:
> > On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:
> >
> > [snip]
> >
> > > > > > > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > > > > > > the function schedule_timeout() is more used than the ssleep() (517
> > > > > > > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> > > > > > >
> > > > > > > The other oddity is that Horms reported on this list that on non Xeon
> > > > > > > CPU the same version of kernel of mine does not present the problem.
> > > > > > >
> > > > > > > I'm getting crazy :-)
> > > > >
> > > > > I've prepared a patch, which reverts the change which was introduced
> > > > > by Nishanth Aravamudan in February.
> > > >
> > > > Was the 100% cpu utilization only occurring on Xeon processors?
> > >
> > > That seems to be the only case where were this problem has been
> > > observed. I don't have such a processor myself, so I haven't actually
> > > been able to produce the problem locally.
> > >
> > > One reason I posted this issue to netdev was to get some more
> > > eyes on the problem as it is puzzling to say the least.
> > >
> > > > Care to try to use msleep_interruptible() instead of ssleep(), as
> > > > opposed to schedule_timeout()?
> > >
> > > I will send a version that does that shortly, Luca, can
> > > you plase check that too?
> >
> > Here is that version of the patch. Nishanth, I take it that I do not
> > need to set TASK_INTERRUPTABLE before calling msleep_interruptible(),
> > please let me know if I am wrong.
>
> Yes, exactly. I'm just trying to narrow it down to see if it's the task
> state that's causing the issue (which, to be honest, doesn't make a lot
> of sense to me -- with ssleep() your load average will go up as the task
> will be UNINTERRUPTIBLE state, but I am not sure why utilisation would
> rise, as you are still sleeping...)
>
> Thanks,
> Nish
>
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: jamal @ 2005-09-26 13:31 UTC (permalink / raw)
To: Michael Bellion; +Cc: linux-kernel, linux-net, netdev
In-Reply-To: <200509261516.16565.mbellion@hipac.org>
On Mon, 2005-26-09 at 15:16 +0200, Michael Bellion wrote:
> Hi,
>
> > Can you post some numbers relative to iptables?
>
> We have some performance tests available at:
> http://www.hipac.org/performance_tests/overview.html
>
> We also have a list of the independent performance tests we know of:
> http://www.hipac.org/performance_tests/independent.html
>
Can you please post something against new kernels you are patching
against _today_? I recall these same graphs from a few years back but
even iptables has improved since.
Any issues you may find can only help you improve.
BTW, your tests were unfair to iptables; you should have had optimized
the rules with the assumption that someone needing that many rules would
probably have needed to do some optimization even with iptables.
Yes, it would only have taken one year to load 256K rules, but it would
have loaded eventually.
> > Some tests with the following parameters would be helpful:
> > - Variable incoming packet rate (in packets per second)
> > - Variable packet sizes
> > - Variable number of users/filters
> > - Effect of adding/removing/modifying policies while under different
> > incoming traffic rates.
>
> Most of this parameters are used in the performance tests above.
>
> The effect of adding/removing/modifying policies while under different
> incoming traffic rates has not been tested in the above tests.
>
> nf-HiPAC is based on a completely dynamic approach.
Very good. Please do more up to date testing and try to include tc
filter as well.
cheers,
jamal
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: Michael Bellion @ 2005-09-26 13:16 UTC (permalink / raw)
To: hadi; +Cc: linux-kernel, linux-net, netdev
In-Reply-To: <1127733492.6215.274.camel@localhost.localdomain>
Hi,
> Can you post some numbers relative to iptables?
We have some performance tests available at:
http://www.hipac.org/performance_tests/overview.html
We also have a list of the independent performance tests we know of:
http://www.hipac.org/performance_tests/independent.html
> Some tests with the following parameters would be helpful:
> - Variable incoming packet rate (in packets per second)
> - Variable packet sizes
> - Variable number of users/filters
> - Effect of adding/removing/modifying policies while under different
> incoming traffic rates.
Most of this parameters are used in the performance tests above.
The effect of adding/removing/modifying policies while under different
incoming traffic rates has not been tested in the above tests.
nf-HiPAC is based on a completely dynamic approach.
This means that the algorithm used in HiPAC makes sure that the lookup data
structure is not rebuild from scratch again as soon as you make a update of
the data structure.
Instead during an update of the policies only the required changes of the
lookup data structure are made. This guaranties that the packet processing is
only affected to the least possible amount during updates.
It would certainly be nice to see some benchmark results for this case.
nf-HiPAC is expected to handle this very well, because it was designed with
this case in mind.
Regards
+---------------------------+
| Michael Bellion |
| <mbellion@hipac.org> |
+---------------------------+
^ permalink raw reply
* Re: ipvs_syncmaster brings cpu to 100%
From: Nishanth Aravamudan @ 2005-09-26 13:11 UTC (permalink / raw)
To: Roger Tsang, Luca Maranzano,
LinuxVirtualServer.org users mailing list., Dave Miller,
Wensong Zhang, Julian Anastasov, netdev
In-Reply-To: <20050926081229.GA23755@verge.net.au>
On 26.09.2005 [17:12:32 +0900], Horms wrote:
> On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:
>
> [snip]
>
> > > > > > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > > > > > the function schedule_timeout() is more used than the ssleep() (517
> > > > > > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> > > > > >
> > > > > > The other oddity is that Horms reported on this list that on non Xeon
> > > > > > CPU the same version of kernel of mine does not present the problem.
> > > > > >
> > > > > > I'm getting crazy :-)
> > > >
> > > > I've prepared a patch, which reverts the change which was introduced
> > > > by Nishanth Aravamudan in February.
> > >
> > > Was the 100% cpu utilization only occurring on Xeon processors?
> >
> > That seems to be the only case where were this problem has been
> > observed. I don't have such a processor myself, so I haven't actually
> > been able to produce the problem locally.
> >
> > One reason I posted this issue to netdev was to get some more
> > eyes on the problem as it is puzzling to say the least.
> >
> > > Care to try to use msleep_interruptible() instead of ssleep(), as
> > > opposed to schedule_timeout()?
> >
> > I will send a version that does that shortly, Luca, can
> > you plase check that too?
>
> Here is that version of the patch. Nishanth, I take it that I do not
> need to set TASK_INTERRUPTABLE before calling msleep_interruptible(),
> please let me know if I am wrong.
Yes, exactly. I'm just trying to narrow it down to see if it's the task
state that's causing the issue (which, to be honest, doesn't make a lot
of sense to me -- with ssleep() your load average will go up as the task
will be UNINTERRUPTIBLE state, but I am not sure why utilisation would
rise, as you are still sleeping...)
Thanks,
Nish
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: jamal @ 2005-09-26 12:40 UTC (permalink / raw)
To: Emmanuel Fleury; +Cc: Michael Bellion, linux-kernel, linux-net, netdev
In-Reply-To: <4337E601.1070208@cs.aau.dk>
On Mon, 2005-26-09 at 14:13 +0200, Emmanuel Fleury wrote:
> That is right, the add/remove/modifying policies was still an issue when
> we stopped working on this. But, I think we can solve this problem by
> working on an incremental compiler. I have to admit that this require a
> proof of concept. This issue is quite critical on our method.
>
I think if you solve the lookup issue (which you seem to have) and this
you will be in good shape.
> > Several comments:
> > - Am i mistaken that your source of data is from somewhere in the
> > backbone? Would it be fair to say that something in the edge would be
> > more appropriate?
>
> The source of the data has been recorded from a backbone but all the
> clients in the experimental setting are replaying part of it and sending
> it to the gigabit switch.
>
Ok, makes sense.
> > - Is tcpreplay the right tool? What does it give you that you cant use
> > a better blaster like pktgen?
>
> The idea was to replay a realistic traffic. So we used a slightly
> modified version of the tcpreplay (I don't remember what modifications
> have been done, my co-author did it). I should definitely take a look at
> pktgen.
>
Both are useful tests. Benchmarking typically doesnt factor in realistic
- the rule of thumb being if you can win a benchmark you can do better.
The exception is when you make complex changes to win a benchmark; there
has to be a balance.
> > If you are going to run these tests in stateless firewalling as you
> > did, please consider using tc filter as well.
>
> The spirit was to test a totally new idea, the plan was not really to
> get integrated (yet) into any tool, so we did it like this. We might
> have been wrong.
I think the methodology was fine enough for what you were doing then;
academic time constraints apply and you can be forgiven for that ;->
Now that you have gone beyond academia and have some company behind your
work, it is only fair to compare against tc filter because it is in the
kernel and being used by some people.
I am willing to help consult for the tc filter side if you need help.
cheers,
jamal
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: Emmanuel Fleury @ 2005-09-26 12:13 UTC (permalink / raw)
To: hadi; +Cc: Michael Bellion, linux-kernel, linux-net, netdev
In-Reply-To: <1127735881.6215.294.camel@localhost.localdomain>
Hi,
jamal wrote:
>
> To repeat the tests i mentioned earlier for clarity:
> a) Variable incoming packet rate (in packets per second)
> b) Variable packet sizes
> c) Variable number of users/filters
> d) Effect of adding/removing/modifying policies while under different
> incoming traffic rates.
>
> You seem to have taken care of most of the variables involved except
> for #d below. If you look at my slides you will see why #d is
> important to have in modern firewalls. I think if you have to first
> compile rules then you will have issues, but it remains to be seen.
That is right, the add/remove/modifying policies was still an issue when
we stopped working on this. But, I think we can solve this problem by
working on an incremental compiler. I have to admit that this require a
proof of concept. This issue is quite critical on our method.
> Several comments:
> - Am i mistaken that your source of data is from somewhere in the
> backbone? Would it be fair to say that something in the edge would be
> more appropriate?
The source of the data has been recorded from a backbone but all the
clients in the experimental setting are replaying part of it and sending
it to the gigabit switch.
> - Your header extraction tool creates "10 sets of rules"; is there a
> reason for the number 10?
No particular reason.
> - Is tcpreplay the right tool? What does it give you that you cant use
> a better blaster like pktgen?
The idea was to replay a realistic traffic. So we used a slightly
modified version of the tcpreplay (I don't remember what modifications
have been done, my co-author did it). I should definitely take a look at
pktgen.
> - I think the blackbox monitor looking at the input vs output tool is
> good. It will be more complete if you can quantify the input rate then
> you can easily quantify output rate.
Interesting suggestion. Thanks.
> - While your results were useful in showing Mbps; they are incomplete
> by not mentioning the packet size. A better metric would have been
> pps. But even then mentioning packet size is also useful.
Right, the packet size was 300bytes (I think this is mentioned in the
text but not highlighted in the figures). I agree with you about pps (we
were young and innocent at the time).
> If you are going to run these tests in stateless firewalling as you
> did, please consider using tc filter as well.
The spirit was to test a totally new idea, the plan was not really to
get integrated (yet) into any tool, so we did it like this. We might
have been wrong.
Thanks for your comments.
Regards
--
Emmanuel Fleury
The highest goal of computer science is to automate that
which can be automated.
-- D. L. VerLee
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: jamal @ 2005-09-26 11:58 UTC (permalink / raw)
To: Emmanuel Fleury; +Cc: Michael Bellion, linux-kernel, linux-net, netdev
In-Reply-To: <4337DA7C.2000804@cs.aau.dk>
On Mon, 2005-26-09 at 13:24 +0200, Emmanuel Fleury wrote:
> Hi,
>
> Did you solved your "size" issues when entering long list of rules ???
>
> I'm still not convinced by your approach. :-/
>
> These experiments have to be updated but can you comment on this:
> http://www.cs.aau.dk/~mixxel/cf/experiments.html
To repeat the tests i mentioned earlier for clarity:
a) Variable incoming packet rate (in packets per second)
b) Variable packet sizes
c) Variable number of users/filters
d) Effect of adding/removing/modifying policies while under different
incoming traffic rates.
You seem to have taken care of most of the variables involved except for
#d below. If you look at my slides you will see why #d is important to
have in modern firewalls. I think if you have to first compile rules
then you will have issues, but it remains to be seen.
Several comments:
- Am i mistaken that your source of data is from somewhere in the
backbone? Would it be fair to say that something in the edge would be
more appropriate?
- Your header extraction tool creates "10 sets of rules"; is there a
reason for the number 10?
- Is tcpreplay the right tool? What does it give you that you cant use a
better blaster like pktgen?
- I think the blackbox monitor looking at the input vs output tool is
good. It will be more complete if you can quantify the input rate then
you can easily quantify output rate.
- While your results were useful in showing Mbps; they are incomplete by
not mentioning the packet size. A better metric would have been pps. But
even then mentioning packet size is also useful.
If you are going to run these tests in stateless firewalling as you did,
please consider using tc filter as well.
cheers,
jamal
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: Emmanuel Fleury @ 2005-09-26 11:24 UTC (permalink / raw)
To: Michael Bellion; +Cc: linux-kernel, linux-net, netdev
In-Reply-To: <200509260445.46740.mbellion@hipac.org>
Hi,
Did you solved your "size" issues when entering long list of rules ???
I'm still not convinced by your approach. :-/
These experiments have to be updated but can you comment on this:
http://www.cs.aau.dk/~mixxel/cf/experiments.html
Regards
--
Emmanuel Fleury
Houston, we've had a problem here.
-- Jack Swigert (Appolo XIII, April 13, 1970)
^ permalink raw reply
* Re: [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: jamal @ 2005-09-26 11:18 UTC (permalink / raw)
To: Michael Bellion; +Cc: linux-kernel, linux-net, netdev
In-Reply-To: <200509260445.46740.mbellion@hipac.org>
On Mon, 2005-26-09 at 04:45 +0200, Michael Bellion wrote:
> Hi
>
> I am happy to announce the release of nf-HiPAC version 0.9.0
>
> During the development of version 0.9.0 everything was ported to Linux kernel
> 2.6 and large parts of the kernel code have been rewritten.
> The kernel patch is now fairly non-intrusive: it only adds one simple function
> to ip_tables.c. The rest of the patch introduces new files to the kernel.
> The new release fixes all known bugs and also introduces some new features.
>
> Since the last release I have become part of MARA Systems AB
> ( http://www.marasystems.com ). MARA Systems AB is now the commercial backer
> of the HiPAC Project and finances it completely. Together MARA Systems and I
> will make sure that HiPAC is actively maintained and further developed under
> the GNU GPL.
>
>
Congratulations to yourself as well as your sponsor. I think this is
useful.
The iptables wrapper is certainly valuable.
Can you post some numbers relative to iptables?
Some tests with the following parameters would be helpful:
- Variable incoming packet rate (in packets per second)
- Variable packet sizes
- Variable number of users/filters
- Effect of adding/removing/modifying policies while under different
incoming traffic rates.
Just even simple non-stateful comparisons like i did with tc over here:
http://www.suug.ch/sucon/04/slides/pkt_cls.pdf
Or even better when you do these tests also try out with tc filter.
cheers,
jamal
^ permalink raw reply
* Re: ipvs_syncmaster brings cpu to 100%
From: Horms @ 2005-09-26 8:12 UTC (permalink / raw)
To: Nishanth Aravamudan, Roger Tsang, Luca Maranzano,
LinuxVirtualServer.org users mailing list., Dave Miller,
Wensong Zhang, Julian Anastasov, netdev
In-Reply-To: <20050926080508.GF11027@verge.net.au>
On Mon, Sep 26, 2005 at 05:05:10PM +0900, Horms wrote:
[snip]
> > > > > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > > > > the function schedule_timeout() is more used than the ssleep() (517
> > > > > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> > > > >
> > > > > The other oddity is that Horms reported on this list that on non Xeon
> > > > > CPU the same version of kernel of mine does not present the problem.
> > > > >
> > > > > I'm getting crazy :-)
> > >
> > > I've prepared a patch, which reverts the change which was introduced
> > > by Nishanth Aravamudan in February.
> >
> > Was the 100% cpu utilization only occurring on Xeon processors?
>
> That seems to be the only case where were this problem has been
> observed. I don't have such a processor myself, so I haven't actually
> been able to produce the problem locally.
>
> One reason I posted this issue to netdev was to get some more
> eyes on the problem as it is puzzling to say the least.
>
> > Care to try to use msleep_interruptible() instead of ssleep(), as
> > opposed to schedule_timeout()?
>
> I will send a version that does that shortly, Luca, can
> you plase check that too?
Here is that version of the patch. Nishanth, I take it that I do not
need to set TASK_INTERRUPTABLE before calling msleep_interruptible(),
please let me know if I am wrong.
Luca, please test.
--
Horms
*UNTESTED*
Use msleep_interruptible() instead of ssleep() in ip_vs_sync daemon,
as the latter seems to cause 100% CPU utilistaion on HT Xeons.
Discussion:
http://archive.linuxvirtualserver.org/html/lvs-users/2005-09/msg00031.html
Reverts:
http://www.kernel.org/git/?p=linux/kernel/git/tglx/history.git;a=commit;h=f8afb60c7537130448cc479d6d8dc9bf4ee06027
Signed-off-by: Horms <horms@verge.net.au>
diff --git a/net/ipv4/ipvs/ip_vs_sync.c b/net/ipv4/ipvs/ip_vs_sync.c
--- a/net/ipv4/ipvs/ip_vs_sync.c
+++ b/net/ipv4/ipvs/ip_vs_sync.c
@@ -655,7 +655,7 @@ static void sync_master_loop(void)
if (stop_master_sync)
break;
- ssleep(1);
+ msleep_interruptible(1000);
}
/* clean up the sync_buff queue */
@@ -712,7 +712,7 @@ static void sync_backup_loop(void)
if (stop_backup_sync)
break;
- ssleep(1);
+ msleep_interruptible(1000);
}
/* release the sending multicast socket */
@@ -824,7 +824,7 @@ static int fork_sync_thread(void *startu
if ((pid = kernel_thread(sync_thread, startup, 0)) < 0) {
IP_VS_ERR("could not create sync_thread due to %d... "
"retrying.\n", pid);
- ssleep(1);
+ msleep_interruptible(1000);
goto repeat;
}
@@ -858,7 +858,7 @@ int start_sync_thread(int state, char *m
if ((pid = kernel_thread(fork_sync_thread, &startup, 0)) < 0) {
IP_VS_ERR("could not create fork_sync_thread due to %d... "
"retrying.\n", pid);
- ssleep(1);
+ msleep_interruptible(1000);
goto repeat;
}
^ permalink raw reply
* Re: ipvs_syncmaster brings cpu to 100%
From: Horms @ 2005-09-26 8:05 UTC (permalink / raw)
To: Nishanth Aravamudan
Cc: Roger Tsang, Luca Maranzano,
LinuxVirtualServer.org users mailing list., Dave Miller,
Wensong Zhang, Julian Anastasov, netdev
In-Reply-To: <20050926043400.GD5079@us.ibm.com>
On Sun, Sep 25, 2005 at 09:34:00PM -0700, Nishanth Aravamudan wrote:
> On 26.09.2005 [12:28:08 +0900], Horms wrote:
> > On Fri, Sep 23, 2005 at 11:15:31AM -0400, Roger Tsang wrote:
> > > As I've said before in this thread, you might want to try changing all the
> > > ssleep() calls to schedule_timeout().
> > >
> > > Roger
> > >
> > >
> > > On 9/22/05, Luca Maranzano <liuk001@gmail.com> wrote:
> > > >
> > > > Hello all,
> > > >
> > > > here again trying to discover the reason ot the CPU hog for
> > > > ipvs_sync{master,backup}.
> > > >
> > > > I've digged in the sources for ip_vs_sync.c and the main differences
> > > > between kernel 2.6.8 and 2.6.12 is the use of ssleep() instead of
> > > > schedule_timeout().
> > > >
> > > > The oddity I've seen is that in the header of both files, the version
> > > > is always like this:
> > > >
> > > > * Version: $Id: ip_vs_sync.c,v 1.13 2003/06/08 09:31:19 wensong Exp $
> > > > *
> > > > * Authors: Wensong Zhang <wensong@linuxvirtualserver.org>
> > > >
> > > > Is Wensong still the maintainer for this code?
> >
> > Yes, although he is kind of quiet.
> >
> > > > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > > > the function schedule_timeout() is more used than the ssleep() (517
> > > > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> > > >
> > > > The other oddity is that Horms reported on this list that on non Xeon
> > > > CPU the same version of kernel of mine does not present the problem.
> > > >
> > > > I'm getting crazy :-)
> >
> > I've prepared a patch, which reverts the change which was introduced
> > by Nishanth Aravamudan in February.
>
> Was the 100% cpu utilization only occurring on Xeon processors?
That seems to be the only case where were this problem has been
observed. I don't have such a processor myself, so I haven't actually
been able to produce the problem locally.
One reason I posted this issue to netdev was to get some more
eyes on the problem as it is puzzling to say the least.
> Care to try to use msleep_interruptible() instead of ssleep(), as
> opposed to schedule_timeout()?
I will send a version that does that shortly, Luca, can
you plase check that too?
> In your patch, you do not need to set the state back to TASK_RUNNING,
> btw.
Thanks, updated patch below.
--
Horms
Use schedule_timeout() instead of ssleep() in ip_vs_sync daemon,
as the latter seems to cause 100% CPU utilistaion on HT Xeons.
Discussion:
http://archive.linuxvirtualserver.org/html/lvs-users/2005-09/msg00031.html
Reverts:
http://www.kernel.org/git/?p=linux/kernel/git/tglx/history.git;a=commit;h=f8afb60c7537130448cc479d6d8dc9bf4ee06027
Signed-off-by: Horms <horms@verge.net.au>
diff --git a/net/ipv4/ipvs/ip_vs_sync.c b/net/ipv4/ipvs/ip_vs_sync.c
--- a/net/ipv4/ipvs/ip_vs_sync.c
+++ b/net/ipv4/ipvs/ip_vs_sync.c
@@ -655,7 +655,8 @@ static void sync_master_loop(void)
if (stop_master_sync)
break;
- ssleep(1);
+ __set_current_state(TASK_INTERRUPTIBLE);
+ schedule_timeout(HZ);
}
/* clean up the sync_buff queue */
@@ -712,7 +713,8 @@ static void sync_backup_loop(void)
if (stop_backup_sync)
break;
- ssleep(1);
+ __set_current_state(TASK_INTERRUPTIBLE);
+ schedule_timeout(HZ);
}
/* release the sending multicast socket */
@@ -824,7 +826,8 @@ static int fork_sync_thread(void *startu
if ((pid = kernel_thread(sync_thread, startup, 0)) < 0) {
IP_VS_ERR("could not create sync_thread due to %d... "
"retrying.\n", pid);
- ssleep(1);
+ __set_current_state(TASK_INTERRUPTIBLE);
+ schedule_timeout(HZ);
goto repeat;
}
@@ -858,7 +861,8 @@ int start_sync_thread(int state, char *m
if ((pid = kernel_thread(fork_sync_thread, &startup, 0)) < 0) {
IP_VS_ERR("could not create fork_sync_thread due to %d... "
"retrying.\n", pid);
- ssleep(1);
+ __set_current_state(TASK_INTERRUPTIBLE);
+ schedule_timeout(HZ);
goto repeat;
}
^ permalink raw reply
* Re: ipvs_syncmaster brings cpu to 100%
From: Nishanth Aravamudan @ 2005-09-26 4:34 UTC (permalink / raw)
To: Roger Tsang, Luca Maranzano,
LinuxVirtualServer.org users mailing list., Dave Miller,
Wensong Zhang, Julian Anastasov, netdev
In-Reply-To: <20050926032807.GI18357@verge.net.au>
On 26.09.2005 [12:28:08 +0900], Horms wrote:
> On Fri, Sep 23, 2005 at 11:15:31AM -0400, Roger Tsang wrote:
> > As I've said before in this thread, you might want to try changing all the
> > ssleep() calls to schedule_timeout().
> >
> > Roger
> >
> >
> > On 9/22/05, Luca Maranzano <liuk001@gmail.com> wrote:
> > >
> > > Hello all,
> > >
> > > here again trying to discover the reason ot the CPU hog for
> > > ipvs_sync{master,backup}.
> > >
> > > I've digged in the sources for ip_vs_sync.c and the main differences
> > > between kernel 2.6.8 and 2.6.12 is the use of ssleep() instead of
> > > schedule_timeout().
> > >
> > > The oddity I've seen is that in the header of both files, the version
> > > is always like this:
> > >
> > > * Version: $Id: ip_vs_sync.c,v 1.13 2003/06/08 09:31:19 wensong Exp $
> > > *
> > > * Authors: Wensong Zhang <wensong@linuxvirtualserver.org>
> > >
> > > Is Wensong still the maintainer for this code?
>
> Yes, although he is kind of quiet.
>
> > > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > > the function schedule_timeout() is more used than the ssleep() (517
> > > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> > >
> > > The other oddity is that Horms reported on this list that on non Xeon
> > > CPU the same version of kernel of mine does not present the problem.
> > >
> > > I'm getting crazy :-)
>
> I've prepared a patch, which reverts the change which was introduced
> by Nishanth Aravamudan in February.
Was the 100% cpu utilization only occurring on Xeon processors?
Care to try to use msleep_interruptible() instead of ssleep(), as
opposed to schedule_timeout()?
In your patch, you do not need to set the state back to TASK_RUNNING,
btw.
Thanks,
Nish
^ permalink raw reply
* Re: ipvs_syncmaster brings cpu to 100%
From: Horms @ 2005-09-26 3:28 UTC (permalink / raw)
To: Roger Tsang, Luca Maranzano,
LinuxVirtualServer.org users mailing list.
Cc: Nishanth Aravamudan, Dave Miller, Wensong Zhang, Julian Anastasov,
netdev
In-Reply-To: <498263350509230815eb08a73@mail.gmail.com>
On Fri, Sep 23, 2005 at 11:15:31AM -0400, Roger Tsang wrote:
> As I've said before in this thread, you might want to try changing all the
> ssleep() calls to schedule_timeout().
>
> Roger
>
>
> On 9/22/05, Luca Maranzano <liuk001@gmail.com> wrote:
> >
> > Hello all,
> >
> > here again trying to discover the reason ot the CPU hog for
> > ipvs_sync{master,backup}.
> >
> > I've digged in the sources for ip_vs_sync.c and the main differences
> > between kernel 2.6.8 and 2.6.12 is the use of ssleep() instead of
> > schedule_timeout().
> >
> > The oddity I've seen is that in the header of both files, the version
> > is always like this:
> >
> > * Version: $Id: ip_vs_sync.c,v 1.13 2003/06/08 09:31:19 wensong Exp $
> > *
> > * Authors: Wensong Zhang <wensong@linuxvirtualserver.org>
> >
> > Is Wensong still the maintainer for this code?
Yes, although he is kind of quiet.
> > Furthermore, if I make an "rgrep" in the source tree of kernel 2.6.12
> > the function schedule_timeout() is more used than the ssleep() (517
> > occurrencies vs. 43), so why in ip_vs_sync.c there was this change?
> >
> > The other oddity is that Horms reported on this list that on non Xeon
> > CPU the same version of kernel of mine does not present the problem.
> >
> > I'm getting crazy :-)
I've prepared a patch, which reverts the change which was introduced
by Nishanth Aravamudan in February.
I have CCed him, Dave Miller, Wensong Zhang, Julian Anastasov, and the
netdev list for comment.
Could intererested parties please test the patch.
Thanks
--
Horms
Use schedule_timeout() instead of ssleep() in ip_vs_sync daemon,
as the latter seems to cause 100% CPU utilistaion on HT Xeons.
Discussion:
http://archive.linuxvirtualserver.org/html/lvs-users/2005-09/msg00031.html
Reverts:
http://www.kernel.org/git/?p=linux/kernel/git/tglx/history.git;a=commit;h=f8afb60c7537130448cc479d6d8dc9bf4ee06027
Signed-off-by: Horms <horms@verge.net.au>
diff --git a/net/ipv4/ipvs/ip_vs_sync.c b/net/ipv4/ipvs/ip_vs_sync.c
--- a/net/ipv4/ipvs/ip_vs_sync.c
+++ b/net/ipv4/ipvs/ip_vs_sync.c
@@ -655,7 +655,9 @@ static void sync_master_loop(void)
if (stop_master_sync)
break;
- ssleep(1);
+ __set_current_state(TASK_INTERRUPTIBLE);
+ schedule_timeout(HZ);
+ __set_current_state(TASK_RUNNING);
}
/* clean up the sync_buff queue */
@@ -712,7 +714,9 @@ static void sync_backup_loop(void)
if (stop_backup_sync)
break;
- ssleep(1);
+ __set_current_state(TASK_INTERRUPTIBLE);
+ schedule_timeout(HZ);
+ __set_current_state(TASK_RUNNING);
}
/* release the sending multicast socket */
@@ -824,7 +828,9 @@ static int fork_sync_thread(void *startu
if ((pid = kernel_thread(sync_thread, startup, 0)) < 0) {
IP_VS_ERR("could not create sync_thread due to %d... "
"retrying.\n", pid);
- ssleep(1);
+ __set_current_state(TASK_INTERRUPTIBLE);
+ schedule_timeout(HZ);
+ __set_current_state(TASK_RUNNING);
goto repeat;
}
@@ -858,7 +864,9 @@ int start_sync_thread(int state, char *m
if ((pid = kernel_thread(fork_sync_thread, &startup, 0)) < 0) {
IP_VS_ERR("could not create fork_sync_thread due to %d... "
"retrying.\n", pid);
- ssleep(1);
+ __set_current_state(TASK_INTERRUPTIBLE);
+ schedule_timeout(HZ);
+ __set_current_state(TASK_RUNNING);
goto repeat;
}
^ permalink raw reply
* [ANNOUNCE] Release of nf-HiPAC 0.9.0
From: Michael Bellion @ 2005-09-26 2:45 UTC (permalink / raw)
To: linux-kernel, linux-net, netdev
Hi
I am happy to announce the release of nf-HiPAC version 0.9.0
During the development of version 0.9.0 everything was ported to Linux kernel
2.6 and large parts of the kernel code have been rewritten.
The kernel patch is now fairly non-intrusive: it only adds one simple function
to ip_tables.c. The rest of the patch introduces new files to the kernel.
The new release fixes all known bugs and also introduces some new features.
Since the last release I have become part of MARA Systems AB
( http://www.marasystems.com ). MARA Systems AB is now the commercial backer
of the HiPAC Project and finances it completely. Together MARA Systems and I
will make sure that HiPAC is actively maintained and further developed under
the GNU GPL.
For all of you who don't know nf-HiPAC yet, here is a short overview:
nf-HiPAC is a full featured packet filter for Linux which demonstrates the
power and flexibility of HiPAC. HiPAC is a novel framework for packet
classification which uses an advanced algorithm to reduce the number of
memory lookups per packet. It is ideal for environments involving large rule
sets and/or high bandwidth networks.
nf-HiPAC provides the same rich feature set as iptables, the popular Linux
packet filter. The complexity of the sophisticated HiPAC packet
classification algorithm is hidden behind an iptables compatible user
interface which renders nf-HiPAC a drop-in replacement for iptables. Thereby,
the iptables' semantics of the rules is preserved, i.e. you can construct your
rules like you are used to. From a user's point of view there is no need to
understand anything about the HiPAC algorithm.
The nf-hipac user space tool is designed to be as compatible as possible to
'iptables -t filter'. It even supports the full power of iptables targets,
matches and stateful packet filtering (connection tracking) besides the native
nf-HiPAC matches. This makes a switch from iptables to nf-HiPAC very easy.
Usually it is sufficient to replace the calls to iptables with calls to
nf-hipac for your filter rules.
Why another packet filter?
Performance:
iptables, like most packet filters, uses a simple packet classification
algorithm which traverses the rules in a chain linearly per packet until a
matching rule is found (or not). Clearly, this approach lacks efficiency.
As networks grow more and more complex and offer a wider bandwidth linear
packet filtering is no longer an option if many rules have to be matched
per packet. Higher bandwidth means more packets per second which leads to
shorter process times per packet. nf-HiPAC outperforms iptables regardless
of the number of rules, i.e. the HiPAC classification engine does not
impose any overhead even for very small rule sets.
Scalability to large rule sets:
The performance of nf-HiPAC is nearly independent of the number of rules.
nf-HiPAC with thousands of rules still outperforms iptables with 20 rules.
Dynamic rule sets:
nf-HiPAC offers fast dynamic rules et updates without stalling packet
classification in contrast to iptables which yields bad update performance
along with stalled packet processing during updates.
More information about the project can be found at: http://www.hipac.org
The releases are published on: http://sourceforge.net/projects/nf-hipac/
Enjoy,
+---------------------------+
| Michael Bellion |
| <mbellion@hipac.org> |
+---------------------------+
^ permalink raw reply
* Re: rwlock recursion on CPU#0, netfilter related?
From: Pekka Pietikainen @ 2005-09-25 20:19 UTC (permalink / raw)
To: Harald Welte; +Cc: netdev
In-Reply-To: <20050925134344.GJ731@sunbeam.de.gnumonks.org>
On Sun, Sep 25, 2005 at 03:43:44PM +0200, Harald Welte wrote:
> 1) how does your kernel .config look like?
http://cvs.fedora.redhat.com/viewcvs/devel/kernel/configs/config-generic?rev=1.60&view=auto
http://cvs.fedora.redhat.com/viewcvs/devel/kernel/configs/config-x86_64-generic?rev=1.16&view=auto
> 2) which modules are loaded
Module Size Used by
w83627hf 46569 0
eeprom 17617 0
i2c_sensor 12225 2 w83627hf,eeprom
i2c_isa 11329 0
rfcomm 61033 0
l2cap 46145 5 rfcomm
bluetooth 73317 4 rfcomm,l2cap
ipv6 325889 16
ppp_synctty 21057 0
ppp_async 22465 1
crc_ccitt 10817 1 ppp_async
ppp_generic 41953 6 ppp_synctty,ppp_async
slhc 16193 1 ppp_generic
ip_conntrack_ftp 82177 0
ipt_ULOG 18913 1
ipt_state 10689 18
ip_conntrack 60053 2 ip_conntrack_ftp,ipt_state
iptable_filter 11969 1
ip_tables 32193 3 ipt_ULOG,ipt_state,iptable_filter
loop 26449 0
video 27977 0
button 16481 0
battery 19657 0
ac 14409 0
ohci1394 46753 0
ieee1394 381273 1 ohci1394
ohci_hcd 33249 0
ehci_hcd 46157 0
parport_pc 40621 0
parport 52557 1 parport_pc
i2c_nforce2 16833 0
i2c_core 34241 5 w83627hf,eeprom,i2c_sensor,i2c_isa,i2c_nforce2
shpchp 108009 0
emu10k1_gp 12865 0
gameport 27089 2 emu10k1_gp
snd_emu10k1 138629 0
snd_rawmidi 39521 1 snd_emu10k1
snd_util_mem 14401 1 snd_emu10k1
snd_hwdep 20321 1 snd_emu10k1
snd_intel8x0 46273 0
snd_ac97_codec 106757 2 snd_emu10k1,snd_intel8x0
snd_seq_dummy 12869 0
snd_seq_oss 47012 0
snd_seq_midi_event 17473 1 snd_seq_oss
snd_seq 74265 5 snd_seq_dummy,snd_seq_oss,snd_seq_midi_event
snd_seq_device 19281 5 snd_emu10k1,snd_rawmidi,snd_seq_dummy,snd_seq_oss,snd_seq
snd_pcm_oss 68465 0
snd_mixer_oss 28225 1 snd_pcm_oss
snd_pcm 115401 4 snd_emu10k1,snd_intel8x0,snd_ac97_codec,snd_pcm_oss
snd_timer 37577 3 snd_emu10k1,snd_seq,snd_pcm
snd 75681 12 snd_emu10k1,snd_rawmidi,snd_hwdep,snd_intel8x0,snd_ac97_codec,snd_seq_oss,snd_seq,snd_seq_device,snd_pcm_oss,snd_mixer_oss,snd_pcm,snd_timer
soundcore 19809 1 snd
snd_page_alloc 21713 3 snd_emu10k1,snd_intel8x0,snd_pcm
r8169 43209 0
forcedeth 30657 0
floppy 77865 0
dm_snapshot 26369 0
dm_zero 10817 0
dm_mirror 32433 0
ext3 154577 3
jbd 76145 1 ext3
dm_mod 73873 7 dm_snapshot,dm_zero,dm_mirror
sata_nv 19141 3
libata 61649 1 sata_nv
sd_mod 29121 4
scsi_mod 167801 2 libata,sd_mod
> 3) how does your ruleset look like?
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -i eth1 -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type echo-request -j ACCEPT
-A RH-Firewall-1-INPUT -p esp -j ACCEPT
-A RH-Firewall-1-INPUT -p ah -j ACCEPT
-A RH-Firewall-1-INPUT -p ipv6 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -j ULOG
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport x -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport y -j ACCEPT
(for a bunch of ports, some with -s sourcenet/24 etc.)
-A RH-Firewall-1-INPUT -j DROP
COMMIT
# Completed on Sun Sep 28 10:37:44 2003
So basically a single-host firewall with ULOG and ftp conntracking being the
only fancy things.
> 4) most importantly, have you enabled CONFIG_IP_NF_CONNTRACK_EVENTS ?
> if yes, please disable, it's broken, a fix has been submitted, but I
> don't know if it has propagated to Linus yet (netdev Message-ID:
> <20050922143515.GD8917@rama.de.gnumonks.org>)
Enabled, so this could be it. But 2.6.14-rc2-git4 did crash too (although
it did take a bit longer for that to happen), and the changelog does state:
commit 1dfbab59498d6f227c91988bab6c71af049a5333
tree 6b20409a232ebe8c37f16d06b3fbcde6bec8f328
parent a82b748930fce0dab22c64075c38c830ae116904
author Harald Welte <laforge@netfilter.org> Thu, 22 Sep 2005 23:46:57 -0700
committer David S. Miller <davem@davemloft.net> Thu, 22 Sep 2005 23:46:57
-0700
[NETFILTER] Fix conntrack event cache deadlock/oops
Which is this patch, right? Will verify whether disabling the option makes any
difference tomorrow, as well as your other recommendations.
> Also, I have that Ping time problem on my x86_64 debian unstable (smp).
> But only in 1 out of ten cases on average (when starting ping, ctrl+c,
> pin, ctrl+c, ...). I've always assumed it's some 64bit problem in
> "ping" itself.
Happens for all packets on the "broken" kernels, and works a-ok (few ms
latencies to the same box) on the 2.6.13-era ones that don't crash.
Could be a different bug, sure.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox