Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.

Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.

[Anthony Liguori]

This is a rather curious path in the code involving rate limiting that 
I'm not all that familiar with.  Does it help is you create the domains 
with a large maxmem than memory?  For instance:

maxmem = 512
memory = 256

Perhaps someone with more experience in the rate limiting code can 
explain why this message is only printed when it's enabled?  To save 
some work, here's the relevant bits from netback.c:net_rx_action():

        if (!xen_feature(XENFEAT_auto_translated_physmap)) {
            /* Memory squeeze? Back off for an arbitrary while. */
            if ((new_mfn = alloc_mfn()) == 0) {
                if ( net_ratelimit() )
                    WPRINTK("Memory squeeze in netback "
                        "driver.\n");
                mod_timer(&net_timer, jiffies + HZ);
                skb_queue_head(&rx_queue, skb);
                break;
            }

Does the rate limiting code cause increase_reservation to fail if you've 
exceeded your limit?

Regards,

Anthony Liguori

Erik Hensema wrote:
> Hi everybody,
>
> When I start more than about 3 domU's the following messages are 
> logged in dom0:
>
> Jun  6 18:30:03 thebe kernel: printk: 18 messages suppressed.
> Jun  6 18:30:03 thebe kernel: xen_net: Memory squeeze in netback 
> driver.
> Jun  6 18:30:03 thebe last message repeated 9 times
>
> This progressively gets worse when I start more and more domU's, up 
> until the point some or all domU's lose their network connectivety.
>
> When I google on the error message, a few pages turn up, but no real 
> sollutions.
>
> I'll try to describe my setup in detail:
>
> the hardware:
>
> AMD Opteron 144
> 4 GB ram
> Dual Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet 
> controllers
>
> It's Aplus hardware from supermicro.
>
> the software:
>
> Suse Linux 10.1, default kernel: 2.6.16.13-4-xen
> Xen version 3.0.2_09656-4 (abuild@suse.de) (gcc version 4.1.0 (SUSE 
> Linux)) Tue May  2 01:39:27 UTC 2006
>
> I've got two bridges: xenbr0 and xenbr1:
> bridge name     bridge id               STP enabled     interfaces
> xenbr0          8000.feffffffffff       no              vif0.0
>                                                         peth0
>                                                         vif1.0
> (all vifX.0)
> xenbr1          8000.feffffffffff       no              vif0.1
>                                                         peth1
>                                                         vif1.1
> (all vifX.1)
>
> So basically I've got two seperate lans, available to all domains. No 
> NAT. One lan is an internal vlan, the other lan is the public 
> internet.
> Each domain runs an iptables firewall, including domain0. The firewall 
> is your basic run-of-the-mill iptables firewall, rules are matching 
> eth0 and eth1, no special modules are used.
>
> domain0's got plenty of free ram when the memory squeeze happens:
> thebe:~ # free -m
>              total       used       free     shared    buffers  cached
> Mem:          1506        646        859          0        290  101
> -/+ buffers/cache:        255       1250
> Swap:         4102          0       4102
>
>   

Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.

[Erik Hensema]

On Tuesday 06 June 2006 23:43, Anthony Liguori wrote:
> This is a rather curious path in the code involving rate limiting
> that I'm not all that familiar with.  Does it help is you create
> the domains with a large maxmem than memory?  For instance:
>
> maxmem = 512
> memory = 256

Yes, it does seem to help. I've recreated almost all domains with 16 
MB 'headroom', now I can create domains up until the point I run out 
of loop devices ;-) (6 domU's running ATM)
I'll reboot tonight (or shutdown all domains and reload the loop 
module) and try to start more domains.


-- 
Erik Hensema (erik@hensema.net)

Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.

[Anthony Liguori]

Erik Hensema wrote:
> On Tuesday 06 June 2006 23:43, Anthony Liguori wrote:
>   
>> This is a rather curious path in the code involving rate limiting
>> that I'm not all that familiar with.  Does it help is you create
>> the domains with a large maxmem than memory?  For instance:
>>
>> maxmem = 512
>> memory = 256
>>     
>
> Yes, it does seem to help. I've recreated almost all domains with 16 
> MB 'headroom', now I can create domains up until the point I run out 
> of loop devices ;-) (6 domU's running ATM)
> I'll reboot tonight (or shutdown all domains and reload the loop 
> module) and try to start more domains.
>   

I suspected it would but I was hoping it wouldn't.  Do this mean that 
the netback driver needs to be able to increase it's reservation?  If 
so, should we be enforcing (in the tools) that memory < maxmem?  Any 
thoughts Keir?

Regards,

Anthony Liguori

Re: [Xen-users] (xen) Possible bug: Memory squeeze in netback driver.

[Keir Fraser]

On 7 Jun 2006, at 21:43, Anthony Liguori wrote:

> I suspected it would but I was hoping it wouldn't.  Do this mean that 
> the netback driver needs to be able to increase it's reservation?  If 
> so, should we be enforcing (in the tools) that memory < maxmem?  Any 
> thoughts Keir?

The kernel should be able to do it for itself. Drivers inform how much 
'headroom' they need via balloon_update_driver_allowance(). Balloon 
driver then strives to keep its memory allocation below maxmem by that 
amount. Possibly the balloon driver needs a kick when allocation fails 
in teh driver, to expand the balloon a little. Another alternative is 
for drivers to incr/decr reservation via another balloon driver 
interface, giving another hook point for the balloon driver to do 
something sensible.

So, in short, tools may be able to screw things for a short while (and 
maybe crash things entirely, if maxmem is reduced drastically with no 
prior warning (update of memory target)). But the balloon driver is 
expected to recover except in that kind of unreasonable circumstance.

  -- Keir