[xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Sander Eikelenboom]

Hi All,

I'm experiencing for what it seems a random freeze with current xen-4.0-testing, pvops dom0 2.6.32.16 kernel, most of the time within 2 days after rebooting.

Symptoms:
- Complete freeze, only power cycle does work.
- No bug output/stacktrace in serial log / on screen.
- Not able to get into hypervisor with ctrl-a (doesn't react to keyboard)
- No info in syslog.

Are there any more boot options I could give a try in the hope it will give some debug output ?



title           xen-4.0.1-rc5-pre.gz / Debian GNU/Linux, 2.6.32.16+xen-2.6.32.x-20100731
root            (hd0,0)
kernel          /xen-4.0.1-rc5-pre.gz dom0_mem=768M loglvl=all loglvl_guest=all com1=115200,8n1 sync_console console_to_ring console=com1,vga iommu=0,verbose,amd_iommu_debug lapic=debug apic_verbosity=debug apic=debug
module          /vmlinuz-2.6.32.16+xen-2.6.32.x-20100731 root=/dev/mapper/serveerstertje-root ro earlyprintk=xen max_loop=255 loop_max_part=63 console=hvc0 xen-pciback.hide=(03:06.0)(04:00.0)(08:00.0)(0a:01.1)(0a:01.2)(0f:00.0) pci=resource_alignment=03:06.0;04:00.0;08:00.0;0a:01.0;0a:01.1;0a:01.2;0f:00.0
module          /initrd.img-2.6.32.16+xen-2.6.32.x-20100731







serveerstertje:~# xm info
host                   : serveerstertje
release                : 2.6.32.16+xen-2.6.32.x-20100731
version                : #1 SMP Sat Jul 31 14:27:35 CEST 2010
machine                : x86_64
nr_cpus                : 6
nr_nodes               : 1
cores_per_socket       : 6
threads_per_core       : 1
cpu_mhz                : 3200
hw_caps                : 178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000037ff:00000000
virt_caps              : hvm
total_memory           : 8191
free_memory            : 6312
node_to_cpu            : node0:0-5
node_to_memory         : node0:6312
node_to_dma32_mem      : node0:2876
max_node_id            : 0
xen_major              : 4
xen_minor              : 0
xen_extra              : .1-rc5-pre
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Sun Jul 25 22:22:43 2010 +0100 21287:e6b5b2cb8146
xen_commandline        : dom0_mem=768M loglvl=all loglvl_guest=all com1=115200,8n1 sync_console console_to_ring console=com1,vga iommu=0,verbose,amd_iommu_debug lapic=debug apic_verbosity=debug apic=debug
cc_compiler            : gcc version 4.3.2 (Debian 4.3.2-1.1)
cc_compile_by          : root
cc_compile_domain      :
cc_compile_date        : Sat Jul 31 12:06:59 CEST 2010
xend_config_format     : 4


commit 78b55f90e72348e231092dbe3e50ac7414b9e1af
Merge: c0a00fb... dee9469...
Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Date:   Wed Jul 28 00:19:39 2010 -0700

    Merge branch 'xen/next-2.6.32' into xen/stable-2.6.32.x

    * xen/next-2.6.32: (24 commits)
      cmpxchg: fix some 32-bit typos
      x86/cmpxchg: fix asm constraints to mention memory modification
      apply_to_page_range: fix compilation warning
      xen/blktap: #if CONFIG_XEN -> #ifdef CONFIG_XEN
      x86/hugepte: use set_pgd for 2-level pagetables
      implement O_NONBLOCK for /proc/xen/xenbus
      xen-pcifront: Remove usage of spin-locks.
      xen-pciback: Redo spinlock usage.
      xen-pcifront: Fix spinlock usage.
      xen-pcifront: Don't race with udev when discovering new devices.
      xen/blktap: make protocol specific usage of shared sring explicit
      xen/netback: make protocol specific usage of shared sring explicit
      xen/rings: make protocol specific usage of shared sring explicit
      xen/rings: make protocol specific usage of shared sring explicit
      xen/netfront: make protocol specific usage of shared sring explicit
      xen/rings: make protocol specific usage of shared sring explicit
      xen/rings: make protocol specific usage of shared sring explicit
      xen/blkback: Flush blkback data when connecting.
      xen: support large numbers of CPUs with vcpu info placement
      rtnetlink: make SR-IOV VF interface symmetric
      ...

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Konrad Rzeszutek Wilk]

On Tue, Aug 03, 2010 at 05:30:57PM +0200, Sander Eikelenboom wrote:
> Hi All,
> 
> I'm experiencing for what it seems a random freeze with current xen-4.0-testing, pvops dom0 2.6.32.16 kernel, most of the time within 2 days after rebooting.
> 
You did not experience the freeze with 2.6.32.15?
> Symptoms:
> - Complete freeze, only power cycle does work.
> - No bug output/stacktrace in serial log / on screen.
> - Not able to get into hypervisor with ctrl-a (doesn't react to keyboard)
> - No info in syslog.
> 
> Are there any more boot options I could give a try in the hope it will give some debug output ?

The Linux kernel has some of those 'DETECT_SPINLOCK_HANG' or
'DETECT_WORK..something' flags. It might be a good idea to compile those
and see when your machine freezes if after 2 minutes the kernel starts
spitting out what is hung. That could give some idea.

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Jeremy Fitzhardinge]

  On 08/03/2010 08:45 AM, Konrad Rzeszutek Wilk wrote:
> On Tue, Aug 03, 2010 at 05:30:57PM +0200, Sander Eikelenboom wrote:
>> Hi All,
>>
>> I'm experiencing for what it seems a random freeze with current xen-4.0-testing, pvops dom0 2.6.32.16 kernel, most of the time within 2 days after rebooting.
>>
> You did not experience the freeze with 2.6.32.15?

There have been a few updates to the .32.16 kernel too (and now its 
.17...).  But it would be very useful to identify which the last working 
kernel was.

>> Symptoms:
>> - Complete freeze, only power cycle does work.
>> - No bug output/stacktrace in serial log / on screen.
>> - Not able to get into hypervisor with ctrl-a (doesn't react to keyboard)
>> - No info in syslog.
>>
>> Are there any more boot options I could give a try in the hope it will give some debug output ?
> The Linux kernel has some of those 'DETECT_SPINLOCK_HANG' or
> 'DETECT_WORK..something' flags. It might be a good idea to compile those
> and see when your machine freezes if after 2 minutes the kernel starts
> spitting out what is hung. That could give some idea.
>

If Xen doesn't respond then it isn't a kernel spinlock problem; it looks 
more system-wide than that.  I notice the kernel command line has lots 
of hidden PCI devices.  Sander, is there any particular activity (esp 
passthrough device activity) which might correspond to the hang?

     J

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Sander Eikelenboom]

Hi Jeremy,

Yes, i have a domU with 2 usb cards passed through, with 2 USB videograbbers attached.
This domain is running Konrad's devel/merge.2.6.35-rc6.t2 tree and some additional patches for the usb3/xhci card, which still give some trouble.

But i didn't expect both dom0 and the hypervisor to freeze as well, and leaving no clues :(

--

Sander


Tuesday, August 3, 2010, 5:51:26 PM, you wrote:

>   On 08/03/2010 08:45 AM, Konrad Rzeszutek Wilk wrote:
>> On Tue, Aug 03, 2010 at 05:30:57PM +0200, Sander Eikelenboom wrote:
>>> Hi All,
>>>
>>> I'm experiencing for what it seems a random freeze with current xen-4.0-testing, pvops dom0 2.6.32.16 kernel, most of the time within 2 days after rebooting.
>>>
>> You did not experience the freeze with 2.6.32.15?

> There have been a few updates to the .32.16 kernel too (and now its 
> .17...).  But it would be very useful to identify which the last working 
> kernel was.

>>> Symptoms:
>>> - Complete freeze, only power cycle does work.
>>> - No bug output/stacktrace in serial log / on screen.
>>> - Not able to get into hypervisor with ctrl-a (doesn't react to keyboard)
>>> - No info in syslog.
>>>
>>> Are there any more boot options I could give a try in the hope it will give some debug output ?
>> The Linux kernel has some of those 'DETECT_SPINLOCK_HANG' or
>> 'DETECT_WORK..something' flags. It might be a good idea to compile those
>> and see when your machine freezes if after 2 minutes the kernel starts
>> spitting out what is hung. That could give some idea.
>>

> If Xen doesn't respond then it isn't a kernel spinlock problem; it looks 
> more system-wide than that.  I notice the kernel command line has lots 
> of hidden PCI devices.  Sander, is there any particular activity (esp 
> passthrough device activity) which might correspond to the hang?

>      J



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Sander Eikelenboom]

Hi Jeremy,

The last kernel that worked is quite hard to say, since the xhci patches depend on a 2.6.35-rc6 kernel, so i took that one from Konrad's tree.
But those pciback changes make it so i have to use a recent dom0 kernel as well (if i recall correct form some spinlock cleanups to work).

--
Sander


Tuesday, August 3, 2010, 5:51:26 PM, you wrote:

>   On 08/03/2010 08:45 AM, Konrad Rzeszutek Wilk wrote:
>> On Tue, Aug 03, 2010 at 05:30:57PM +0200, Sander Eikelenboom wrote:
>>> Hi All,
>>>
>>> I'm experiencing for what it seems a random freeze with current xen-4.0-testing, pvops dom0 2.6.32.16 kernel, most of the time within 2 days after rebooting.
>>>
>> You did not experience the freeze with 2.6.32.15?

> There have been a few updates to the .32.16 kernel too (and now its 
> .17...).  But it would be very useful to identify which the last working 
> kernel was.

>>> Symptoms:
>>> - Complete freeze, only power cycle does work.
>>> - No bug output/stacktrace in serial log / on screen.
>>> - Not able to get into hypervisor with ctrl-a (doesn't react to keyboard)
>>> - No info in syslog.
>>>
>>> Are there any more boot options I could give a try in the hope it will give some debug output ?
>> The Linux kernel has some of those 'DETECT_SPINLOCK_HANG' or
>> 'DETECT_WORK..something' flags. It might be a good idea to compile those
>> and see when your machine freezes if after 2 minutes the kernel starts
>> spitting out what is hung. That could give some idea.
>>

> If Xen doesn't respond then it isn't a kernel spinlock problem; it looks 
> more system-wide than that.  I notice the kernel command line has lots 
> of hidden PCI devices.  Sander, is there any particular activity (esp 
> passthrough device activity) which might correspond to the hang?

>      J



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Sander Eikelenboom]

Hi Konrad/Jeremy,

I have tested the last 2 days with the vm's with passthroughed devices shutdown, and no freeze so far.
I'm running now with one of the vm's that runs an old 2.6.33 kernel from an old tree from Konrad together with some hacked up patches for xhci/usb3 support.
That seems to be running fine for some time now (although not a full 2 days yet).

So my other vm seems to cause the freeze.

- This one uses the devel/merge.2.6.35-rc6.t2 as domU kernel, i think i should try an older version of pci-front/xen-swiotlb perhaps.
- It has both a usb2 and usb3 controller passed through, but the xhci module has much changed since the hacked up patches from the kernel in de working domU vm
- Most probably the drivers for the videograbbers will have changed

So i suspect:
   - newer pci-front / xen-swiotlb
   - xhci/usb3 driver
   - drivers videograbber

Most probable would be a roque dma transfer that can't be catched by xen / pciback I guess, and therefore would be hard to debug ?

--
Sander



Tuesday, August 3, 2010, 5:51:26 PM, you wrote:

>   On 08/03/2010 08:45 AM, Konrad Rzeszutek Wilk wrote:
>> On Tue, Aug 03, 2010 at 05:30:57PM +0200, Sander Eikelenboom wrote:
>>> Hi All,
>>>
>>> I'm experiencing for what it seems a random freeze with current xen-4.0-testing, pvops dom0 2.6.32.16 kernel, most of the time within 2 days after rebooting.
>>>
>> You did not experience the freeze with 2.6.32.15?

> There have been a few updates to the .32.16 kernel too (and now its 
> .17...).  But it would be very useful to identify which the last working 
> kernel was.

>>> Symptoms:
>>> - Complete freeze, only power cycle does work.
>>> - No bug output/stacktrace in serial log / on screen.
>>> - Not able to get into hypervisor with ctrl-a (doesn't react to keyboard)
>>> - No info in syslog.
>>>
>>> Are there any more boot options I could give a try in the hope it will give some debug output ?
>> The Linux kernel has some of those 'DETECT_SPINLOCK_HANG' or
>> 'DETECT_WORK..something' flags. It might be a good idea to compile those
>> and see when your machine freezes if after 2 minutes the kernel starts
>> spitting out what is hung. That could give some idea.
>>

> If Xen doesn't respond then it isn't a kernel spinlock problem; it looks 
> more system-wide than that.  I notice the kernel command line has lots 
> of hidden PCI devices.  Sander, is there any particular activity (esp 
> passthrough device activity) which might correspond to the hang?

>      J



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Konrad Rzeszutek Wilk]

On Thu, Aug 05, 2010 at 11:48:44AM +0200, Sander Eikelenboom wrote:
> Hi Konrad/Jeremy,
> 
> I have tested the last 2 days with the vm's with passthroughed devices shutdown, and no freeze so far.
> I'm running now with one of the vm's that runs an old 2.6.33 kernel from an old tree from Konrad together with some hacked up patches for xhci/usb3 support.
> That seems to be running fine for some time now (although not a full 2 days yet).
> 
> So my other vm seems to cause the freeze.
> 
> - This one uses the devel/merge.2.6.35-rc6.t2 as domU kernel, i think i should try an older version of pci-front/xen-swiotlb perhaps.
> - It has both a usb2 and usb3 controller passed through, but the xhci module has much changed since the hacked up patches from the kernel in de working domU vm
> - Most probably the drivers for the videograbbers will have changed
> 
> So i suspect:
>    - newer pci-front / xen-swiotlb
>    - xhci/usb3 driver
>    - drivers videograbber
> 
> Most probable would be a roque dma transfer that can't be catched by xen / pciback I guess, and therefore would be hard to debug ?

The SWIOTLB "brains" by themselves haven't changed since the
uhh...2.6.33. The code internals that just got Ack-ed upstream looks quite
similar to the one that Jeremy carries in xen/stable-2.6.32.x. The
outside plumbing parts are the ones that changed.

The fixes in the pci-front, well, most of those are "burocractic" in
nature - set the ownership to this, make hotplug work, etc. The big
fixes were the MSI/MSI-X ones but those were big news a couple of months
ago (and I think that was when 2.6.34 came out).

The videograbber (vl4) stack trace you sent to me some time ago looked
liked a mutex was held for a very very long time... which I wonder if
that is the cmpxch compiler bug that has hit some folks. Are you using
Debian?

But we can do something easy. I can rebase my 2.6.33 kernel with the
latest Xen-SWIOTLB/SWIOTLB engine + Xen PCI front, and we can eliminate the
SWIOTLB/PCIfront being at fault here.. Let me do that if your  2.6.33
VM guest is running fine for the last two days.

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Sander Eikelenboom]

Hi Konrad,

Thx for your response, and i saw Linus pulled the swiotlb code .. way to go !

Thursday, August 5, 2010, 4:52:14 PM, you wrote:

> On Thu, Aug 05, 2010 at 11:48:44AM +0200, Sander Eikelenboom wrote:
>> Hi Konrad/Jeremy,
>> 
>> I have tested the last 2 days with the vm's with passthroughed devices shutdown, and no freeze so far.
>> I'm running now with one of the vm's that runs an old 2.6.33 kernel from an old tree from Konrad together with some hacked up patches for xhci/usb3 support.
>> That seems to be running fine for some time now (although not a full 2 days yet).
>> 
>> So my other vm seems to cause the freeze.
>> 
>> - This one uses the devel/merge.2.6.35-rc6.t2 as domU kernel, i think i should try an older version of pci-front/xen-swiotlb perhaps.
>> - It has both a usb2 and usb3 controller passed through, but the xhci module has much changed since the hacked up patches from the kernel in de working domU vm
>> - Most probably the drivers for the videograbbers will have changed
>> 
>> So i suspect:
>>    - newer pci-front / xen-swiotlb
>>    - xhci/usb3 driver
>>    - drivers videograbber
>> 
>> Most probable would be a roque dma transfer that can't be catched by xen / pciback I guess, and therefore would be hard to debug ?

> The SWIOTLB "brains" by themselves haven't changed since the
> uhh...2.6.33. The code internals that just got Ack-ed upstream looks quite
> similar to the one that Jeremy carries in xen/stable-2.6.32.x. The
> outside plumbing parts are the ones that changed.

> The fixes in the pci-front, well, most of those are "burocractic" in
> nature - set the ownership to this, make hotplug work, etc. The big
> fixes were the MSI/MSI-X ones but those were big news a couple of months
> ago (and I think that was when 2.6.34 came out).

> The videograbber (vl4) stack trace you sent to me some time ago looked
> liked a mutex was held for a very very long time... which I wonder if
> that is the cmpxch compiler bug that has hit some folks. Are you using
> Debian?

Yes i'm using Debian, i saw that bug fix too, but since Jeremy didn't include it in stable yet i also didn't :-)
Well you gave me a pointer here, looking again it seems to hang on the device on the usb2 controller and not the usb3.
So to rule out the usb3 stuff i will drop that usb2 controller and see if that works. If so, it must be a problem in the driver.
Since that grabber + usb2 controller worked for quite a while grabbing perfectly.


> But we can do something easy. I can rebase my 2.6.33 kernel with the
> latest Xen-SWIOTLB/SWIOTLB engine + Xen PCI front, and we can eliminate the
> SWIOTLB/PCIfront being at fault here.. Let me do that if your  2.6.33
> VM guest is running fine for the last two days.

I will first try the above, if that doesn't work out,  i will try the 2.6.33 again for longer and report back !

Thx Again !

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Jeremy Fitzhardinge]

  On 08/05/2010 07:52 AM, Konrad Rzeszutek Wilk wrote:
> The videograbber (vl4) stack trace you sent to me some time ago looked
> liked a mutex was held for a very very long time... which I wonder if
> that is the cmpxch compiler bug that has hit some folks. Are you using
> Debian?

The symptom of that bug is that gcc doesn't see any writes to a static 
variable, so it puts it in a RO section, causing faults.  So while I 
guess its within the realm of possibilities that this is a different 
manifestation of the same bug, it doesn't seem likely.

     J

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Sander Eikelenboom]

Hi Konrad,

Hmm it seems that 2.6.33 tree does seem to work for 1 VM with a videograbber, but doesn't for the VM which seem to cause the freeze.
It does spit out some stacktraces after a while of not functioning, with since is OOM i will be something else caused by the fall out and not anywhere near the root cause.
Although this at least didn't freeze the complete system :-)
I will try some more configurations to see if i can find a pattern somehow ...

--
Sander

[ 1269.032133] submit of urb 0 failed (error=-90)
[ 1274.153341] motion: page allocation failure. order:6, mode:0xd4
[ 1274.153375] Pid: 1884, comm: motion Not tainted 2.6.33 #5
[ 1274.153391] Call Trace:
[ 1274.153416]  [<ffffffff810e4665>] __alloc_pages_nodemask+0x5b2/0x62b
[ 1274.153440]  [<ffffffff810338b9>] ? xen_force_evtchn_callback+0xd/0xf
[ 1274.153461]  [<ffffffff810e46f5>] __get_free_pages+0x17/0x5f
[ 1274.153483]  [<ffffffff8128042e>] xen_swiotlb_alloc_coherent+0x3c/0xe2
[ 1274.153507]  [<ffffffff81410931>] hcd_buffer_alloc+0xfa/0x11f
[ 1274.153527]  [<ffffffff81403e0c>] usb_buffer_alloc+0x17/0x1d
[ 1274.153562]  [<ffffffffa003f39e>] em28xx_init_isoc+0x16a/0x32b [em28xx]
[ 1274.153585]  [<ffffffff815ec0b9>] ? __down_read+0x47/0xed
[ 1274.153613]  [<ffffffffa003a4ac>] buffer_prepare+0xd7/0x10d [em28xx]
[ 1274.153639]  [<ffffffffa0016dac>] videobuf_qbuf+0x308/0x3f4 [videobuf_core]
[ 1274.153667]  [<ffffffffa0039cb3>] vidioc_qbuf+0x35/0x3a [em28xx]
[ 1274.153697]  [<ffffffffa0028229>] __video_do_ioctl+0x11ab/0x373b [videodev]
[ 1274.153720]  [<ffffffff814b51cd>] ? sock_def_readable+0x54/0x5f
[ 1274.153743]  [<ffffffff81541f65>] ? unix_dgram_sendmsg+0x3f1/0x43e
[ 1274.153764]  [<ffffffff810313b5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
[ 1274.153793]  [<ffffffffa0039c7e>] ? vidioc_qbuf+0x0/0x3a [em28xx]
[ 1274.153814]  [<ffffffff814b208b>] ? sock_sendmsg+0xa3/0xbc
[ 1274.153837]  [<ffffffff8123349b>] ? avc_has_perm+0x4e/0x60
[ 1274.153855]  [<ffffffff810338b9>] ? xen_force_evtchn_callback+0xd/0xf
[ 1274.153880]  [<ffffffffa002aab1>] video_ioctl2+0x2f8/0x3af [videodev]
[ 1274.153901]  [<ffffffff810357df>] ? __switch_to+0x265/0x277
[ 1274.153924]  [<ffffffffa0026122>] v4l2_ioctl+0x38/0x3a [videodev]
[ 1274.153944]  [<ffffffff8111ec90>] vfs_ioctl+0x72/0x9e
[ 1274.153961]  [<ffffffff8111f1d7>] do_vfs_ioctl+0x4a0/0x4e1
[ 1274.153980]  [<ffffffff8111f26d>] sys_ioctl+0x55/0x77
[ 1274.154000]  [<ffffffff81112e6a>] ? sys_write+0x60/0x70
[ 1274.154009]  [<ffffffff81036cc2>] system_call_fastpath+0x16/0x1b
[ 1274.154126] Mem-Info:
[ 1274.154138] DMA per-cpu:
[ 1274.154151] CPU    0: hi:    0, btch:   1 usd:   0
[ 1274.154165] CPU    1: hi:    0, btch:   1 usd:   0
[ 1274.154180] DMA32 per-cpu:
[ 1274.154202] CPU    0: hi:  186, btch:  31 usd:   0
[ 1274.154220] CPU    1: hi:  186, btch:  31 usd:  78
[ 1274.154241] active_anon:248 inactive_anon:326 isolated_anon:0
[ 1274.154244]  active_file:132 inactive_file:105 isolated_file:41
[ 1274.154247]  unevictable:0 dirty:0 writeback:19 unstable:0
[ 1274.154250]  free:1309 slab_reclaimable:642 slab_unreclaimable:3111
[ 1274.154254]  mapped:100846 shmem:4 pagetables:1187 bounce:0
[ 1274.154313] DMA free:2036kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:24kB active_file:20kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14752kB mlocked:0kB dirty:0kB writeback:0kB mapped:12804kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 1274.154375] lowmem_reserve[]: 0 489 489 489
[ 1274.154415] DMA32 free:3200kB min:2788kB low:3484kB high:4180kB active_anon:992kB inactive_anon:1280kB active_file:508kB inactive_file:420kB unevictable:0kB isolated(anon):0kB isolated(file):164kB present:500960kB mlocked:0kB dirty:0kB writeback:76kB mapped:390580kB shmem:16kB slab_reclaimable:2552kB slab_unreclaimable:12404kB kernel_stack:592kB pagetables:4724kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
[ 1274.154481] lowmem_reserve[]: 0 0 0 0
[ 1274.154508] DMA: 7*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2036kB
[ 1274.154571] DMA32: 409*4kB 33*8kB 2*16kB 0*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3212kB
[ 1274.154634] 429 total pagecache pages
[ 1274.154646] 161 pages in swap cache
[ 1274.154658] Swap cache stats: add 344422, delete 344260, find 99167/143153
[ 1274.154673] Free swap  = 476756kB
[ 1274.154684] Total swap = 524280kB
[ 1274.160880] 131072 pages RAM
[ 1274.160902] 21934 pages reserved
[ 1274.160914] 101195 pages shared
[ 1274.160925] 6309 pages non-shared
[ 1274.160963] unable to allocate 185088 bytes for transfer buffer 4
[ 1287.634682] motion invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
[ 1287.634719] motion cpuset=/ mems_allowed=0




Thursday, August 5, 2010, 4:52:14 PM, you wrote:

> On Thu, Aug 05, 2010 at 11:48:44AM +0200, Sander Eikelenboom wrote:
>> Hi Konrad/Jeremy,
>> 
>> I have tested the last 2 days with the vm's with passthroughed devices shutdown, and no freeze so far.
>> I'm running now with one of the vm's that runs an old 2.6.33 kernel from an old tree from Konrad together with some hacked up patches for xhci/usb3 support.
>> That seems to be running fine for some time now (although not a full 2 days yet).
>> 
>> So my other vm seems to cause the freeze.
>> 
>> - This one uses the devel/merge.2.6.35-rc6.t2 as domU kernel, i think i should try an older version of pci-front/xen-swiotlb perhaps.
>> - It has both a usb2 and usb3 controller passed through, but the xhci module has much changed since the hacked up patches from the kernel in de working domU vm
>> - Most probably the drivers for the videograbbers will have changed
>> 
>> So i suspect:
>>    - newer pci-front / xen-swiotlb
>>    - xhci/usb3 driver
>>    - drivers videograbber
>> 
>> Most probable would be a roque dma transfer that can't be catched by xen / pciback I guess, and therefore would be hard to debug ?

> The SWIOTLB "brains" by themselves haven't changed since the
> uhh...2.6.33. The code internals that just got Ack-ed upstream looks quite
> similar to the one that Jeremy carries in xen/stable-2.6.32.x. The
> outside plumbing parts are the ones that changed.

> The fixes in the pci-front, well, most of those are "burocractic" in
> nature - set the ownership to this, make hotplug work, etc. The big
> fixes were the MSI/MSI-X ones but those were big news a couple of months
> ago (and I think that was when 2.6.34 came out).

> The videograbber (vl4) stack trace you sent to me some time ago looked
> liked a mutex was held for a very very long time... which I wonder if
> that is the cmpxch compiler bug that has hit some folks. Are you using
> Debian?

> But we can do something easy. I can rebase my 2.6.33 kernel with the
> latest Xen-SWIOTLB/SWIOTLB engine + Xen PCI front, and we can eliminate the
> SWIOTLB/PCIfront being at fault here.. Let me do that if your  2.6.33
> VM guest is running fine for the last two days.




-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Konrad Rzeszutek Wilk]

On Fri, Aug 06, 2010 at 11:21:11AM +0200, Sander Eikelenboom wrote:
> Hi Konrad,
> 
> Hmm it seems that 2.6.33 tree does seem to work for 1 VM with a videograbber, but doesn't for the VM which seem to cause the freeze.
> It does spit out some stacktraces after a while of not functioning, with since is OOM i will be something else caused by the fall out and not anywhere near the root cause.
> Although this at least didn't freeze the complete system :-)
> I will try some more configurations to see if i can find a pattern somehow ...
> 
> --
> Sander
> 
> [ 1269.032133] submit of urb 0 failed (error=-90)
> [ 1274.153341] motion: page allocation failure. order:6, mode:0xd4

That is a 256kB request for memery.
> [ 1274.153375] Pid: 1884, comm: motion Not tainted 2.6.33 #5
> [ 1274.153391] Call Trace:
> [ 1274.153416]  [<ffffffff810e4665>] __alloc_pages_nodemask+0x5b2/0x62b
> [ 1274.153440]  [<ffffffff810338b9>] ? xen_force_evtchn_callback+0xd/0xf
> [ 1274.153461]  [<ffffffff810e46f5>] __get_free_pages+0x17/0x5f
> [ 1274.153483]  [<ffffffff8128042e>] xen_swiotlb_alloc_coherent+0x3c/0xe2
> [ 1274.153507]  [<ffffffff81410931>] hcd_buffer_alloc+0xfa/0x11f
> [ 1274.153527]  [<ffffffff81403e0c>] usb_buffer_alloc+0x17/0x1d
> [ 1274.153562]  [<ffffffffa003f39e>] em28xx_init_isoc+0x16a/0x32b [em28xx]
> [ 1274.153585]  [<ffffffff815ec0b9>] ? __down_read+0x47/0xed
> [ 1274.153613]  [<ffffffffa003a4ac>] buffer_prepare+0xd7/0x10d [em28xx]
> [ 1274.153639]  [<ffffffffa0016dac>] videobuf_qbuf+0x308/0x3f4 [videobuf_core]
> [ 1274.153667]  [<ffffffffa0039cb3>] vidioc_qbuf+0x35/0x3a [em28xx]
> [ 1274.153697]  [<ffffffffa0028229>] __video_do_ioctl+0x11ab/0x373b [videodev]
> [ 1274.153720]  [<ffffffff814b51cd>] ? sock_def_readable+0x54/0x5f
> [ 1274.153743]  [<ffffffff81541f65>] ? unix_dgram_sendmsg+0x3f1/0x43e
> [ 1274.153764]  [<ffffffff810313b5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
> [ 1274.153793]  [<ffffffffa0039c7e>] ? vidioc_qbuf+0x0/0x3a [em28xx]
> [ 1274.153814]  [<ffffffff814b208b>] ? sock_sendmsg+0xa3/0xbc
> [ 1274.153837]  [<ffffffff8123349b>] ? avc_has_perm+0x4e/0x60
> [ 1274.153855]  [<ffffffff810338b9>] ? xen_force_evtchn_callback+0xd/0xf
> [ 1274.153880]  [<ffffffffa002aab1>] video_ioctl2+0x2f8/0x3af [videodev]
> [ 1274.153901]  [<ffffffff810357df>] ? __switch_to+0x265/0x277
> [ 1274.153924]  [<ffffffffa0026122>] v4l2_ioctl+0x38/0x3a [videodev]
> [ 1274.153944]  [<ffffffff8111ec90>] vfs_ioctl+0x72/0x9e
> [ 1274.153961]  [<ffffffff8111f1d7>] do_vfs_ioctl+0x4a0/0x4e1
> [ 1274.153980]  [<ffffffff8111f26d>] sys_ioctl+0x55/0x77
> [ 1274.154000]  [<ffffffff81112e6a>] ? sys_write+0x60/0x70
> [ 1274.154009]  [<ffffffff81036cc2>] system_call_fastpath+0x16/0x1b
> [ 1274.154126] Mem-Info:
> [ 1274.154138] DMA per-cpu:
> [ 1274.154151] CPU    0: hi:    0, btch:   1 usd:   0
> [ 1274.154165] CPU    1: hi:    0, btch:   1 usd:   0
> [ 1274.154180] DMA32 per-cpu:
> [ 1274.154202] CPU    0: hi:  186, btch:  31 usd:   0
> [ 1274.154220] CPU    1: hi:  186, btch:  31 usd:  78
> [ 1274.154241] active_anon:248 inactive_anon:326 isolated_anon:0
> [ 1274.154244]  active_file:132 inactive_file:105 isolated_file:41
> [ 1274.154247]  unevictable:0 dirty:0 writeback:19 unstable:0
> [ 1274.154250]  free:1309 slab_reclaimable:642 slab_unreclaimable:3111
> [ 1274.154254]  mapped:100846 shmem:4 pagetables:1187 bounce:0
> [ 1274.154313] DMA free:2036kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:24kB active_file:20kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14752kB mlocked:0kB dirty:0kB writeback:0kB mapped:12804kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> [ 1274.154375] lowmem_reserve[]: 0 489 489 489
> [ 1274.154415] DMA32 free:3200kB min:2788kB low:3484kB high:4180kB active_anon:992kB inactive_anon:1280kB active_file:508kB inactive_file:420kB unevictable:0kB isolated(anon):0kB isolated(file):164kB present:500960kB mlocked:0kB dirty:0kB writeback:76kB mapped:390580kB shmem:16kB slab_reclaimable:2552kB slab_unreclaimable:12404kB kernel_stack:592kB pagetables:4724kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
> [ 1274.154481] lowmem_reserve[]: 0 0 0 0
> [ 1274.154508] DMA: 7*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2036kB
> [ 1274.154571] DMA32: 409*4kB 33*8kB 2*16kB 0*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3212kB
> [ 1274.154634] 429 total pagecache pages
> [ 1274.154646] 161 pages in swap cache
> [ 1274.154658] Swap cache stats: add 344422, delete 344260, find 99167/143153
> [ 1274.154673] Free swap  = 476756kB
> [ 1274.154684] Total swap = 524280kB
> [ 1274.160880] 131072 pages RAM
> [ 1274.160902] 21934 pages reserved
> [ 1274.160914] 101195 pages shared
> [ 1274.160925] 6309 pages non-shared
> [ 1274.160963] unable to allocate 185088 bytes for transfer buffer 4

Though here it says it is 185 kbytes. Hmm.. You got 3MB in DMA32 and 2MB
in DMA so that should be enough.

I am not that familiar with the VM, so the instinctive thing I can think
of is to raise the amount of memory your guest has from the 512MB to
768MB. Does 'proc/meminfo' when this happens show you an excedingly
small amount of MemFree?

> [ 1287.634682] motion invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
> [ 1287.634719] motion cpuset=/ mems_allowed=0
> 
> 
> 
> 
> Thursday, August 5, 2010, 4:52:14 PM, you wrote:
> 
> > On Thu, Aug 05, 2010 at 11:48:44AM +0200, Sander Eikelenboom wrote:
> >> Hi Konrad/Jeremy,
> >> 
> >> I have tested the last 2 days with the vm's with passthroughed devices shutdown, and no freeze so far.
> >> I'm running now with one of the vm's that runs an old 2.6.33 kernel from an old tree from Konrad together with some hacked up patches for xhci/usb3 support.
> >> That seems to be running fine for some time now (although not a full 2 days yet).
> >> 
> >> So my other vm seems to cause the freeze.
> >> 
> >> - This one uses the devel/merge.2.6.35-rc6.t2 as domU kernel, i think i should try an older version of pci-front/xen-swiotlb perhaps.
> >> - It has both a usb2 and usb3 controller passed through, but the xhci module has much changed since the hacked up patches from the kernel in de working domU vm
> >> - Most probably the drivers for the videograbbers will have changed
> >> 
> >> So i suspect:
> >>    - newer pci-front / xen-swiotlb
> >>    - xhci/usb3 driver
> >>    - drivers videograbber
> >> 
> >> Most probable would be a roque dma transfer that can't be catched by xen / pciback I guess, and therefore would be hard to debug ?
> 
> > The SWIOTLB "brains" by themselves haven't changed since the
> > uhh...2.6.33. The code internals that just got Ack-ed upstream looks quite
> > similar to the one that Jeremy carries in xen/stable-2.6.32.x. The
> > outside plumbing parts are the ones that changed.
> 
> > The fixes in the pci-front, well, most of those are "burocractic" in
> > nature - set the ownership to this, make hotplug work, etc. The big
> > fixes were the MSI/MSI-X ones but those were big news a couple of months
> > ago (and I think that was when 2.6.34 came out).
> 
> > The videograbber (vl4) stack trace you sent to me some time ago looked
> > liked a mutex was held for a very very long time... which I wonder if
> > that is the cmpxch compiler bug that has hit some folks. Are you using
> > Debian?
> 
> > But we can do something easy. I can rebase my 2.6.33 kernel with the
> > latest Xen-SWIOTLB/SWIOTLB engine + Xen PCI front, and we can eliminate the
> > SWIOTLB/PCIfront being at fault here.. Let me do that if your  2.6.33
> > VM guest is running fine for the last two days.
> 
> 
> 
> 
> -- 
> Best regards,
>  Sander                            mailto:linux@eikelenboom.it

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Jeremy Fitzhardinge]

  On 08/06/2010 08:17 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Aug 06, 2010 at 11:21:11AM +0200, Sander Eikelenboom wrote:
>> Hi Konrad,
>>
>> Hmm it seems that 2.6.33 tree does seem to work for 1 VM with a videograbber, but doesn't for the VM which seem to cause the freeze.
>> It does spit out some stacktraces after a while of not functioning, with since is OOM i will be something else caused by the fall out and not anywhere near the root cause.
>> Although this at least didn't freeze the complete system :-)
>> I will try some more configurations to see if i can find a pattern somehow ...
>>
>> --
>> Sander
>>
>> [ 1269.032133] submit of urb 0 failed (error=-90)
>> [ 1274.153341] motion: page allocation failure. order:6, mode:0xd4
> That is a 256kB request for memery.
>> [ 1274.153375] Pid: 1884, comm: motion Not tainted 2.6.33 #5
>> [ 1274.153391] Call Trace:
>> [ 1274.153416]  [<ffffffff810e4665>] __alloc_pages_nodemask+0x5b2/0x62b
>> [ 1274.153440]  [<ffffffff810338b9>] ? xen_force_evtchn_callback+0xd/0xf
>> [ 1274.153461]  [<ffffffff810e46f5>] __get_free_pages+0x17/0x5f
>> [ 1274.153483]  [<ffffffff8128042e>] xen_swiotlb_alloc_coherent+0x3c/0xe2
>> [ 1274.153507]  [<ffffffff81410931>] hcd_buffer_alloc+0xfa/0x11f
>> [ 1274.153527]  [<ffffffff81403e0c>] usb_buffer_alloc+0x17/0x1d
>> [ 1274.153562]  [<ffffffffa003f39e>] em28xx_init_isoc+0x16a/0x32b [em28xx]
>> [ 1274.153585]  [<ffffffff815ec0b9>] ? __down_read+0x47/0xed
>> [ 1274.153613]  [<ffffffffa003a4ac>] buffer_prepare+0xd7/0x10d [em28xx]
>> [ 1274.153639]  [<ffffffffa0016dac>] videobuf_qbuf+0x308/0x3f4 [videobuf_core]
>> [ 1274.153667]  [<ffffffffa0039cb3>] vidioc_qbuf+0x35/0x3a [em28xx]
>> [ 1274.153697]  [<ffffffffa0028229>] __video_do_ioctl+0x11ab/0x373b [videodev]
>> [ 1274.153720]  [<ffffffff814b51cd>] ? sock_def_readable+0x54/0x5f
>> [ 1274.153743]  [<ffffffff81541f65>] ? unix_dgram_sendmsg+0x3f1/0x43e
>> [ 1274.153764]  [<ffffffff810313b5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
>> [ 1274.153793]  [<ffffffffa0039c7e>] ? vidioc_qbuf+0x0/0x3a [em28xx]
>> [ 1274.153814]  [<ffffffff814b208b>] ? sock_sendmsg+0xa3/0xbc
>> [ 1274.153837]  [<ffffffff8123349b>] ? avc_has_perm+0x4e/0x60
>> [ 1274.153855]  [<ffffffff810338b9>] ? xen_force_evtchn_callback+0xd/0xf
>> [ 1274.153880]  [<ffffffffa002aab1>] video_ioctl2+0x2f8/0x3af [videodev]
>> [ 1274.153901]  [<ffffffff810357df>] ? __switch_to+0x265/0x277
>> [ 1274.153924]  [<ffffffffa0026122>] v4l2_ioctl+0x38/0x3a [videodev]
>> [ 1274.153944]  [<ffffffff8111ec90>] vfs_ioctl+0x72/0x9e
>> [ 1274.153961]  [<ffffffff8111f1d7>] do_vfs_ioctl+0x4a0/0x4e1
>> [ 1274.153980]  [<ffffffff8111f26d>] sys_ioctl+0x55/0x77
>> [ 1274.154000]  [<ffffffff81112e6a>] ? sys_write+0x60/0x70
>> [ 1274.154009]  [<ffffffff81036cc2>] system_call_fastpath+0x16/0x1b
>> [ 1274.154126] Mem-Info:
>> [ 1274.154138] DMA per-cpu:
>> [ 1274.154151] CPU    0: hi:    0, btch:   1 usd:   0
>> [ 1274.154165] CPU    1: hi:    0, btch:   1 usd:   0
>> [ 1274.154180] DMA32 per-cpu:
>> [ 1274.154202] CPU    0: hi:  186, btch:  31 usd:   0
>> [ 1274.154220] CPU    1: hi:  186, btch:  31 usd:  78
>> [ 1274.154241] active_anon:248 inactive_anon:326 isolated_anon:0
>> [ 1274.154244]  active_file:132 inactive_file:105 isolated_file:41
>> [ 1274.154247]  unevictable:0 dirty:0 writeback:19 unstable:0
>> [ 1274.154250]  free:1309 slab_reclaimable:642 slab_unreclaimable:3111
>> [ 1274.154254]  mapped:100846 shmem:4 pagetables:1187 bounce:0
>> [ 1274.154313] DMA free:2036kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:24kB active_file:20kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14752kB mlocked:0kB dirty:0kB writeback:0kB mapped:12804kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
>> [ 1274.154375] lowmem_reserve[]: 0 489 489 489
>> [ 1274.154415] DMA32 free:3200kB min:2788kB low:3484kB high:4180kB active_anon:992kB inactive_anon:1280kB active_file:508kB inactive_file:420kB unevictable:0kB isolated(anon):0kB isolated(file):164kB present:500960kB mlocked:0kB dirty:0kB writeback:76kB mapped:390580kB shmem:16kB slab_reclaimable:2552kB slab_unreclaimable:12404kB kernel_stack:592kB pagetables:4724kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
>> [ 1274.154481] lowmem_reserve[]: 0 0 0 0
>> [ 1274.154508] DMA: 7*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2036kB
>> [ 1274.154571] DMA32: 409*4kB 33*8kB 2*16kB 0*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3212kB
>> [ 1274.154634] 429 total pagecache pages
>> [ 1274.154646] 161 pages in swap cache
>> [ 1274.154658] Swap cache stats: add 344422, delete 344260, find 99167/143153
>> [ 1274.154673] Free swap  = 476756kB
>> [ 1274.154684] Total swap = 524280kB
>> [ 1274.160880] 131072 pages RAM
>> [ 1274.160902] 21934 pages reserved
>> [ 1274.160914] 101195 pages shared
>> [ 1274.160925] 6309 pages non-shared
>> [ 1274.160963] unable to allocate 185088 bytes for transfer buffer 4
> Though here it says it is 185 kbytes. Hmm.. You got 3MB in DMA32 and 2MB
> in DMA so that should be enough.
>
> I am not that familiar with the VM, so the instinctive thing I can think
> of is to raise the amount of memory your guest has from the 512MB to
> 768MB. Does 'proc/meminfo' when this happens show you an excedingly
> small amount of MemFree?

Memory allocations are rounded up to the next order, so 185k -> 256k.  
It's also a contiguous allocation, so it needs to find 64 contiguous 
pages, which is pretty much impossible in a system which has been 
running for a while.

     J

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Sander Eikelenboom]

Hi Konrad/Jeremy,

Previously i had a working setup, with the same VM i'm using now, but quite a few things have changed:

- Another motherboard
- Another processor
- Another USB controller (now usb3 pci-e instead of usb2 pci)

The freezes seem to be related to the USB3, but i'm not quite sure, since this also can give another workload.
What i have tried:
- latest xen-4.0-testing compiled with debug=y
- for dom0: latest 2.6.32 pvops stable kernel from Jeremy
- for dom0: latest 2.6.32 xen-next kernel from Jeremy
- added some kernel debug options(apart from some noise about hardirq's this doesn't seem to deliver a lot)
- for domU: 2.6.35-rc6 tree (devel/2.6.25-rc6-t2) from Konrad's tree.
- I also tried running without msi

It allways freezes at some point, but the time when seems to vary, although most of the time within a day.

- Done another memtest to be sure it isn't faulty memory, cooling is on max and temperatures are good, so that seems to be ruled out as well.


- I tried using the IOMMU, this should rule out DMA one should say, but it also froze with the IOMMU enabled and working, and again nothing in serial log :-(

The things i'm about to try after i have finished some backups are:
- xen-unstable
- running the VM as a HVM



What i'm wondering about is:

- Could the 4GB memory barrier still be a problem ? The machine has 8GB, and the domain normally would be started as one of the last, which totals up to around the 4GB of domains running.
  This night i let it run with only the troublesome pv domain, and it seems to work so far.
- Is there a way to force a domain to live underneath the 4GB or any other thing i could try out (besides ripping the ram out of the machine)
- Are there any other things that could prevent a full freeze by making things more strict cq provide addiontal debug info ?

--
Sander

Re: [xen-4.0.1-rc5-pre] [pvops 2.6.32.16] Complete freeze within 2 days, no info in serial log

[Sander Eikelenboom]

[-- Attachment #1: Type: text/plain, Size: 22490 bytes --]

Hi Konrad,

This time the grabbing application became an hung application again in the VM, it seems you are right, available mem is down to 0.
It always used to work with 512mb assigned to the domain.

Most probably a bug in the xhci code i assume ?

Attached: Some hopefully relevant data from /proc


--
Sander



Aug  8 20:16:17 security kernel: [  721.555787] BUG: soft lockup - CPU#0 stuck for 82s! [kmemleak:374]
Aug  8 20:16:17 security kernel: [  721.555790] Modules linked in: fuse saa7115 em28xx v4l2_common videodev v4l1_compat v4l2_compat_ioctl32 videobuf_vmalloc videobuf_core tveeprom evdev i2c_core pcspkr thermal_sys [last unloaded: scsi_wait_scan]
Aug  8 20:16:17 security kernel: [  721.555814] CPU 0 
Aug  8 20:16:17 security kernel: [  721.555816] Modules linked in: fuse saa7115 em28xx v4l2_common videodev v4l1_compat v4l2_compat_ioctl32 videobuf_vmalloc videobuf_core tveeprom evdev i2c_core pcspkr thermal_sys [last unloaded: scsi_wait_scan]
Aug  8 20:16:17 security kernel: [  721.555838] 
Aug  8 20:16:17 security kernel: [  721.555841] Pid: 374, comm: kmemleak Not tainted 2.6.35-rc6+xen-2.6.35-rc6-xen-isoc-20100808-l3-mutex-dma-ed+ #7 /
Aug  8 20:16:17 security kernel: [  721.555847] RIP: e030:[<ffffffff81006318>]  [<ffffffff81006318>] xen_restore_fl_direct+0x18/0x1b
Aug  8 20:16:17 security kernel: [  721.555858] RSP: e02b:ffff88001d8abe40  EFLAGS: 00000246
Aug  8 20:16:17 security kernel: [  721.555861] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88001f7626d0
Aug  8 20:16:17 security kernel: [  721.555865] RDX: 0000000000000000 RSI: 0000000000000200 RDI: 0000000000000200
Aug  8 20:16:17 security kernel: [  721.555869] RBP: 0000000000000001 R08: fffc000000000000 R09: ffff88001d8abdb0
Aug  8 20:16:17 security kernel: [  721.555873] R10: 000000000000000c R11: ffffea00002fdef8 R12: 0000000000000200
Aug  8 20:16:17 security kernel: [  721.555877] R13: 0000000000000000 R14: ffffea00002fdf01 R15: 0000000000000001
Aug  8 20:16:17 security kernel: [  721.555886] FS:  00007fc794dfc910(0000) GS:ffff880002ced000(0000) knlGS:0000000000000000
Aug  8 20:16:17 security kernel: [  721.555891] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug  8 20:16:17 security kernel: [  721.555894] CR2: 0000000001840078 CR3: 000000001e40b000 CR4: 0000000000000660
Aug  8 20:16:17 security kernel: [  721.559780] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug  8 20:16:17 security kernel: [  721.559780] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug  8 20:16:17 security kernel: [  721.559780] Process kmemleak (pid: 374, threadinfo ffff88001d8aa000, task ffff88001fd918d0)
Aug  8 20:16:17 security kernel: [  721.559780] Stack:
Aug  8 20:16:17 security kernel: [  721.559780]  ffffffff8142d77e ffffffff810c638e 0000000000000000 ffff8800126e0260
Aug  8 20:16:17 security kernel: [  721.559780] <0> ffffea00002fdf00 ffffffff810c6967 ffff8800149b02b0 ffffea00002fded0
Aug  8 20:16:17 security kernel: [  721.559780] <0> 000000000000dad6 ffffea00002fdf08 0000000000020000 0000000000000000
Aug  8 20:16:17 security kernel: [  721.559780] Call Trace:
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff8142d77e>] ? _raw_read_unlock_irqrestore+0xd/0xe
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c638e>] ? find_and_get_object+0x4a/0x75
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c6967>] ? scan_block+0x4a/0xf7
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c6ce9>] ? kmemleak_scan+0x1a2/0x3e9
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c737c>] ? kmemleak_scan_thread+0x0/0x9b
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c737c>] ? kmemleak_scan_thread+0x0/0x9b
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c73d5>] ? kmemleak_scan_thread+0x59/0x9b
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff81054051>] ? kthread+0x79/0x81
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810094e4>] ? kernel_thread_helper+0x4/0x10
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810088e3>] ? int_ret_from_sys_call+0x7/0x1b
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff8142dadd>] ? retint_restore_args+0x5/0x6
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810094e0>] ? kernel_thread_helper+0x0/0x10
Aug  8 20:16:17 security kernel: [  721.559780] Code: 44 00 00 65 f6 04 25 21 b0 00 00 ff 0f 94 c4 00 e4 c3 90 66 f7 c7 00 02 65 0f 94 04 25 21 b0 00 00 65 66 83 3c 25 20 b0 00 00 01 <74> 05 e8 01 00 00 00 c3 50 51 52 56 57 41 50 41 51 41 52 41 53 
Aug  8 20:16:17 security kernel: [  721.559780] Call Trace:
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff8142d77e>] ? _raw_read_unlock_irqrestore+0xd/0xe
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c638e>] ? find_and_get_object+0x4a/0x75
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c6967>] ? scan_block+0x4a/0xf7
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c6ce9>] ? kmemleak_scan+0x1a2/0x3e9
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c737c>] ? kmemleak_scan_thread+0x0/0x9b
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c737c>] ? kmemleak_scan_thread+0x0/0x9b
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810c73d5>] ? kmemleak_scan_thread+0x59/0x9b
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff81054051>] ? kthread+0x79/0x81
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810094e4>] ? kernel_thread_helper+0x4/0x10
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810088e3>] ? int_ret_from_sys_call+0x7/0x1b
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff8142dadd>] ? retint_restore_args+0x5/0x6
Aug  8 20:16:17 security kernel: [  721.559780]  [<ffffffff810094e0>] ? kernel_thread_helper+0x0/0x10
Aug  8 20:16:19 security kernel: [  724.187104] kmemleak: 5 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
Aug  8 20:16:46 security motion: [0] Thread 1 - Watchdog timeout, trying to do a graceful restart
Aug  8 20:17:01 security /USR/SBIN/CRON[1865]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug  8 20:17:46 security motion: [0] Thread 1 - Watchdog timeout, did NOT restart graceful,killing it!
Aug  8 20:17:46 security motion: [0] Calling vid_close() from motion_cleanup
Aug  8 20:17:46 security motion: [0] Closing video device /dev/kworld
Aug  8 20:20:17 security kernel: [  961.780121] INFO: task motion:1257 blocked for more than 120 seconds.
Aug  8 20:20:17 security kernel: [  961.780155] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug  8 20:20:17 security kernel: [  961.780177] motion        D ffff88001ef60bc0     0  1257      1 0x00000000
Aug  8 20:20:17 security kernel: [  961.780207]  ffff88001fd155d0 0000000000000282 ffffffff81005cc5 00000000000145c0
Aug  8 20:20:17 security kernel: [  961.780243]  ffff88001e41dfd8 ffff88001e41dfd8 ffff88001ef60930 00000000000145c0
Aug  8 20:20:17 security kernel: [  961.780278]  00000000000145c0 00000000000145c0 ffff88001ef60930 0000000000000000
Aug  8 20:20:17 security kernel: [  961.780313] Call Trace:
Aug  8 20:20:17 security kernel: [  961.780337]  [<ffffffff81005cc5>] ? xen_force_evtchn_callback+0x9/0xa
Aug  8 20:20:17 security kernel: [  961.780365]  [<ffffffffa002eaa6>] ? video_ioctl2+0x0/0x32e [videodev]
Aug  8 20:20:17 security kernel: [  961.780388]  [<ffffffff8142c544>] ? __mutex_lock_slowpath+0x12f/0x22c
Aug  8 20:20:17 security kernel: [  961.780409]  [<ffffffff8142c64a>] ? mutex_lock+0x9/0x1e
Aug  8 20:20:17 security kernel: [  961.780430]  [<ffffffffa0017e58>] ? videobuf_streamoff+0x13/0x35 [videobuf_core]
Aug  8 20:20:17 security kernel: [  961.780454]  [<ffffffff81005cc5>] ? xen_force_evtchn_callback+0x9/0xa
Aug  8 20:20:17 security kernel: [  961.780478]  [<ffffffffa003d573>] ? vidioc_streamoff+0x7e/0xb5 [em28xx]
Aug  8 20:20:17 security kernel: [  961.780500]  [<ffffffffa002c5fe>] ? __video_do_ioctl+0x181f/0x3cc7 [videodev]
Aug  8 20:20:17 security kernel: [  961.780523]  [<ffffffff8100631f>] ? xen_restore_fl_direct_end+0x0/0x1
Aug  8 20:20:17 security kernel: [  961.780544]  [<ffffffff8142d714>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Aug  8 20:20:17 security kernel: [  961.780564]  [<ffffffff813941dd>] ? sock_def_readable+0x3b/0x5d
Aug  8 20:20:17 security kernel: [  961.780585]  [<ffffffff814043a6>] ? unix_dgram_sendmsg+0x428/0x4b2
Aug  8 20:20:17 security kernel: [  961.780606]  [<ffffffff810058fa>] ? xen_set_pte_at+0x196/0x1b6
Aug  8 20:20:17 security kernel: [  961.780625]  [<ffffffff810036bd>] ? __raw_callee_save_xen_make_pte+0x11/0x1e
Aug  8 20:20:17 security kernel: [  961.780648]  [<ffffffff8139115e>] ? sock_sendmsg+0xd1/0xec
Aug  8 20:20:17 security kernel: [  961.780669]  [<ffffffff810b0b00>] ? __do_fault+0x40f/0x44a
Aug  8 20:20:17 security kernel: [  961.780689]  [<ffffffff81005cc5>] ? xen_force_evtchn_callback+0x9/0xa
Aug  8 20:20:17 security kernel: [  961.780709]  [<ffffffff81006332>] ? check_events+0x12/0x20
Aug  8 20:20:17 security kernel: [  961.780730]  [<ffffffffa002ed38>] ? video_ioctl2+0x292/0x32e [videodev]
Aug  8 20:20:17 security kernel: [  961.780750]  [<ffffffff81002616>] ? xen_write_msr_safe+0x5d/0x79
Aug  8 20:20:17 security kernel: [  961.780770]  [<ffffffff81007337>] ? __switch_to+0x1b3/0x2a4
Aug  8 20:20:17 security kernel: [  961.780790]  [<ffffffff8100622a>] ? xen_sched_clock+0xf/0x8c
Aug  8 20:20:17 security kernel: [  961.780810]  [<ffffffff81005cc5>] ? xen_force_evtchn_callback+0x9/0xa
Aug  8 20:20:17 security kernel: [  961.780830]  [<ffffffff81006332>] ? check_events+0x12/0x20
Aug  8 20:20:17 security kernel: [  961.780850]  [<ffffffffa002a10b>] ? v4l2_ioctl+0x38/0x3a [videodev]
Aug  8 20:20:17 security kernel: [  961.780870]  [<ffffffff810d54be>] ? vfs_ioctl+0x69/0x92
Aug  8 20:20:17 security kernel: [  961.780889]  [<ffffffff810d596e>] ? do_vfs_ioctl+0x411/0x43c
Aug  8 20:20:17 security kernel: [  961.780909]  [<ffffffff810c96b4>] ? vfs_write+0x134/0x169
Aug  8 20:20:17 security kernel: [  961.780928]  [<ffffffff810d59ea>] ? sys_ioctl+0x51/0x70
Aug  8 20:20:17 security kernel: [  961.780947]  [<ffffffff810086c2>] ? system_call_fastpath+0x16/0x1b
Aug  8 20:22:17 security kernel: [ 1081.780140] INFO: task motion:1257 blocked for more than 120 seconds.
Aug  8 20:22:17 security kernel: [ 1081.780172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug  8 20:22:17 security kernel: [ 1081.780194] motion        D ffff88001ef60bc0     0  1257      1 0x00000000
Aug  8 20:22:17 security kernel: [ 1081.780224]  ffff88001fd155d0 0000000000000282 ffffffff81005cc5 00000000000145c0
Aug  8 20:22:17 security kernel: [ 1081.780261]  ffff88001e41dfd8 ffff88001e41dfd8 ffff88001ef60930 00000000000145c0
Aug  8 20:22:17 security kernel: [ 1081.780295]  00000000000145c0 00000000000145c0 ffff88001ef60930 0000000000000000
Aug  8 20:22:17 security kernel: [ 1081.780330] Call Trace:
Aug  8 20:22:17 security kernel: [ 1081.780355]  [<ffffffff81005cc5>] ? xen_force_evtchn_callback+0x9/0xa
Aug  8 20:22:17 security kernel: [ 1081.780382]  [<ffffffffa002eaa6>] ? video_ioctl2+0x0/0x32e [videodev]
Aug  8 20:22:17 security kernel: [ 1081.780405]  [<ffffffff8142c544>] ? __mutex_lock_slowpath+0x12f/0x22c
Aug  8 20:22:17 security kernel: [ 1081.780426]  [<ffffffff8142c64a>] ? mutex_lock+0x9/0x1e
Aug  8 20:22:17 security kernel: [ 1081.780447]  [<ffffffffa0017e58>] ? videobuf_streamoff+0x13/0x35 [videobuf_core]
Aug  8 20:22:17 security kernel: [ 1081.780471]  [<ffffffff81005cc5>] ? xen_force_evtchn_callback+0x9/0xa
Aug  8 20:22:17 security kernel: [ 1081.780495]  [<ffffffffa003d573>] ? vidioc_streamoff+0x7e/0xb5 [em28xx]
Aug  8 20:22:17 security kernel: [ 1081.780517]  [<ffffffffa002c5fe>] ? __video_do_ioctl+0x181f/0x3cc7 [videodev]
Aug  8 20:22:17 security kernel: [ 1081.780540]  [<ffffffff8100631f>] ? xen_restore_fl_direct_end+0x0/0x1
Aug  8 20:22:17 security kernel: [ 1081.780561]  [<ffffffff8142d714>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Aug  8 20:22:17 security kernel: [ 1081.780581]  [<ffffffff813941dd>] ? sock_def_readable+0x3b/0x5d
Aug  8 20:22:17 security kernel: [ 1081.780602]  [<ffffffff814043a6>] ? unix_dgram_sendmsg+0x428/0x4b2
Aug  8 20:22:17 security kernel: [ 1081.780622]  [<ffffffff810058fa>] ? xen_set_pte_at+0x196/0x1b6
Aug  8 20:22:17 security kernel: [ 1081.780642]  [<ffffffff810036bd>] ? __raw_callee_save_xen_make_pte+0x11/0x1e
Aug  8 20:22:17 security kernel: [ 1081.780666]  [<ffffffff8139115e>] ? sock_sendmsg+0xd1/0xec
Aug  8 20:22:17 security kernel: [ 1081.780686]  [<ffffffff810b0b00>] ? __do_fault+0x40f/0x44a
Aug  8 20:22:17 security kernel: [ 1081.780706]  [<ffffffff81005cc5>] ? xen_force_evtchn_callback+0x9/0xa
Aug  8 20:22:17 security kernel: [ 1081.780726]  [<ffffffff81006332>] ? check_events+0x12/0x20
Aug  8 20:22:17 security kernel: [ 1081.780747]  [<ffffffffa002ed38>] ? video_ioctl2+0x292/0x32e [videodev]
Aug  8 20:22:17 security kernel: [ 1081.780767]  [<ffffffff81002616>] ? xen_write_msr_safe+0x5d/0x79
Aug  8 20:22:17 security kernel: [ 1081.780787]  [<ffffffff81007337>] ? __switch_to+0x1b3/0x2a4
Aug  8 20:22:17 security kernel: [ 1081.780806]  [<ffffffff8100622a>] ? xen_sched_clock+0xf/0x8c
Aug  8 20:22:17 security kernel: [ 1081.780826]  [<ffffffff81005cc5>] ? xen_force_evtchn_callback+0x9/0xa
Aug  8 20:22:17 security kernel: [ 1081.780847]  [<ffffffff81006332>] ? check_events+0x12/0x20
Aug  8 20:22:17 security kernel: [ 1081.780866]  [<ffffffffa002a10b>] ? v4l2_ioctl+0x38/0x3a [videodev]
Aug  8 20:22:17 security kernel: [ 1081.780886]  [<ffffffff810d54be>] ? vfs_ioctl+0x69/0x92
Aug  8 20:22:17 security kernel: [ 1081.780905]  [<ffffffff810d596e>] ? do_vfs_ioctl+0x411/0x43c
Aug  8 20:22:17 security kernel: [ 1081.780925]  [<ffffffff810c96b4>] ? vfs_write+0x134/0x169
Aug  8 20:22:17 security kernel: [ 1081.780943]  [<ffffffff810d59ea>] ? sys_ioctl+0x51/0x70
Aug  8 20:22:17 security kernel: [ 1081.780961]  [<ffffffff810086c2>] ? system_call_fastpath+0x16/0x1b






Friday, August 6, 2010, 5:17:43 PM, you wrote:

> On Fri, Aug 06, 2010 at 11:21:11AM +0200, Sander Eikelenboom wrote:
>> Hi Konrad,
>> 
>> Hmm it seems that 2.6.33 tree does seem to work for 1 VM with a videograbber, but doesn't for the VM which seem to cause the freeze.
>> It does spit out some stacktraces after a while of not functioning, with since is OOM i will be something else caused by the fall out and not anywhere near the root cause.
>> Although this at least didn't freeze the complete system :-)
>> I will try some more configurations to see if i can find a pattern somehow ...
>> 
>> --
>> Sander
>> 
>> [ 1269.032133] submit of urb 0 failed (error=-90)
>> [ 1274.153341] motion: page allocation failure. order:6, mode:0xd4

> That is a 256kB request for memery.
>> [ 1274.153375] Pid: 1884, comm: motion Not tainted 2.6.33 #5
>> [ 1274.153391] Call Trace:
>> [ 1274.153416]  [<ffffffff810e4665>] __alloc_pages_nodemask+0x5b2/0x62b
>> [ 1274.153440]  [<ffffffff810338b9>] ? xen_force_evtchn_callback+0xd/0xf
>> [ 1274.153461]  [<ffffffff810e46f5>] __get_free_pages+0x17/0x5f
>> [ 1274.153483]  [<ffffffff8128042e>] xen_swiotlb_alloc_coherent+0x3c/0xe2
>> [ 1274.153507]  [<ffffffff81410931>] hcd_buffer_alloc+0xfa/0x11f
>> [ 1274.153527]  [<ffffffff81403e0c>] usb_buffer_alloc+0x17/0x1d
>> [ 1274.153562]  [<ffffffffa003f39e>] em28xx_init_isoc+0x16a/0x32b [em28xx]
>> [ 1274.153585]  [<ffffffff815ec0b9>] ? __down_read+0x47/0xed
>> [ 1274.153613]  [<ffffffffa003a4ac>] buffer_prepare+0xd7/0x10d [em28xx]
>> [ 1274.153639]  [<ffffffffa0016dac>] videobuf_qbuf+0x308/0x3f4 [videobuf_core]
>> [ 1274.153667]  [<ffffffffa0039cb3>] vidioc_qbuf+0x35/0x3a [em28xx]
>> [ 1274.153697]  [<ffffffffa0028229>] __video_do_ioctl+0x11ab/0x373b [videodev]
>> [ 1274.153720]  [<ffffffff814b51cd>] ? sock_def_readable+0x54/0x5f
>> [ 1274.153743]  [<ffffffff81541f65>] ? unix_dgram_sendmsg+0x3f1/0x43e
>> [ 1274.153764]  [<ffffffff810313b5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
>> [ 1274.153793]  [<ffffffffa0039c7e>] ? vidioc_qbuf+0x0/0x3a [em28xx]
>> [ 1274.153814]  [<ffffffff814b208b>] ? sock_sendmsg+0xa3/0xbc
>> [ 1274.153837]  [<ffffffff8123349b>] ? avc_has_perm+0x4e/0x60
>> [ 1274.153855]  [<ffffffff810338b9>] ? xen_force_evtchn_callback+0xd/0xf
>> [ 1274.153880]  [<ffffffffa002aab1>] video_ioctl2+0x2f8/0x3af [videodev]
>> [ 1274.153901]  [<ffffffff810357df>] ? __switch_to+0x265/0x277
>> [ 1274.153924]  [<ffffffffa0026122>] v4l2_ioctl+0x38/0x3a [videodev]
>> [ 1274.153944]  [<ffffffff8111ec90>] vfs_ioctl+0x72/0x9e
>> [ 1274.153961]  [<ffffffff8111f1d7>] do_vfs_ioctl+0x4a0/0x4e1
>> [ 1274.153980]  [<ffffffff8111f26d>] sys_ioctl+0x55/0x77
>> [ 1274.154000]  [<ffffffff81112e6a>] ? sys_write+0x60/0x70
>> [ 1274.154009]  [<ffffffff81036cc2>] system_call_fastpath+0x16/0x1b
>> [ 1274.154126] Mem-Info:
>> [ 1274.154138] DMA per-cpu:
>> [ 1274.154151] CPU    0: hi:    0, btch:   1 usd:   0
>> [ 1274.154165] CPU    1: hi:    0, btch:   1 usd:   0
>> [ 1274.154180] DMA32 per-cpu:
>> [ 1274.154202] CPU    0: hi:  186, btch:  31 usd:   0
>> [ 1274.154220] CPU    1: hi:  186, btch:  31 usd:  78
>> [ 1274.154241] active_anon:248 inactive_anon:326 isolated_anon:0
>> [ 1274.154244]  active_file:132 inactive_file:105 isolated_file:41
>> [ 1274.154247]  unevictable:0 dirty:0 writeback:19 unstable:0
>> [ 1274.154250]  free:1309 slab_reclaimable:642 slab_unreclaimable:3111
>> [ 1274.154254]  mapped:100846 shmem:4 pagetables:1187 bounce:0
>> [ 1274.154313] DMA free:2036kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:24kB active_file:20kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14752kB mlocked:0kB dirty:0kB writeback:0kB mapped:12804kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:24kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
>> [ 1274.154375] lowmem_reserve[]: 0 489 489 489
>> [ 1274.154415] DMA32 free:3200kB min:2788kB low:3484kB high:4180kB active_anon:992kB inactive_anon:1280kB active_file:508kB inactive_file:420kB unevictable:0kB isolated(anon):0kB isolated(file):164kB present:500960kB mlocked:0kB dirty:0kB writeback:76kB mapped:390580kB shmem:16kB slab_reclaimable:2552kB slab_unreclaimable:12404kB kernel_stack:592kB pagetables:4724kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
>> [ 1274.154481] lowmem_reserve[]: 0 0 0 0
>> [ 1274.154508] DMA: 7*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2036kB
>> [ 1274.154571] DMA32: 409*4kB 33*8kB 2*16kB 0*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3212kB
>> [ 1274.154634] 429 total pagecache pages
>> [ 1274.154646] 161 pages in swap cache
>> [ 1274.154658] Swap cache stats: add 344422, delete 344260, find 99167/143153
>> [ 1274.154673] Free swap  = 476756kB
>> [ 1274.154684] Total swap = 524280kB
>> [ 1274.160880] 131072 pages RAM
>> [ 1274.160902] 21934 pages reserved
>> [ 1274.160914] 101195 pages shared
>> [ 1274.160925] 6309 pages non-shared
>> [ 1274.160963] unable to allocate 185088 bytes for transfer buffer 4

> Though here it says it is 185 kbytes. Hmm.. You got 3MB in DMA32 and 2MB
> in DMA so that should be enough.

> I am not that familiar with the VM, so the instinctive thing I can think
> of is to raise the amount of memory your guest has from the 512MB to
> 768MB. Does 'proc/meminfo' when this happens show you an excedingly
> small amount of MemFree?

>> [ 1287.634682] motion invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
>> [ 1287.634719] motion cpuset=/ mems_allowed=0
>> 
>> 
>> 
>> 
>> Thursday, August 5, 2010, 4:52:14 PM, you wrote:
>> 
>> > On Thu, Aug 05, 2010 at 11:48:44AM +0200, Sander Eikelenboom wrote:
>> >> Hi Konrad/Jeremy,
>> >> 
>> >> I have tested the last 2 days with the vm's with passthroughed devices shutdown, and no freeze so far.
>> >> I'm running now with one of the vm's that runs an old 2.6.33 kernel from an old tree from Konrad together with some hacked up patches for xhci/usb3 support.
>> >> That seems to be running fine for some time now (although not a full 2 days yet).
>> >> 
>> >> So my other vm seems to cause the freeze.
>> >> 
>> >> - This one uses the devel/merge.2.6.35-rc6.t2 as domU kernel, i think i should try an older version of pci-front/xen-swiotlb perhaps.
>> >> - It has both a usb2 and usb3 controller passed through, but the xhci module has much changed since the hacked up patches from the kernel in de working domU vm
>> >> - Most probably the drivers for the videograbbers will have changed
>> >> 
>> >> So i suspect:
>> >>    - newer pci-front / xen-swiotlb
>> >>    - xhci/usb3 driver
>> >>    - drivers videograbber
>> >> 
>> >> Most probable would be a roque dma transfer that can't be catched by xen / pciback I guess, and therefore would be hard to debug ?
>> 
>> > The SWIOTLB "brains" by themselves haven't changed since the
>> > uhh...2.6.33. The code internals that just got Ack-ed upstream looks quite
>> > similar to the one that Jeremy carries in xen/stable-2.6.32.x. The
>> > outside plumbing parts are the ones that changed.
>> 
>> > The fixes in the pci-front, well, most of those are "burocractic" in
>> > nature - set the ownership to this, make hotplug work, etc. The big
>> > fixes were the MSI/MSI-X ones but those were big news a couple of months
>> > ago (and I think that was when 2.6.34 came out).
>> 
>> > The videograbber (vl4) stack trace you sent to me some time ago looked
>> > liked a mutex was held for a very very long time... which I wonder if
>> > that is the cmpxch compiler bug that has hit some folks. Are you using
>> > Debian?
>> 
>> > But we can do something easy. I can rebase my 2.6.33 kernel with the
>> > latest Xen-SWIOTLB/SWIOTLB engine + Xen PCI front, and we can eliminate the
>> > SWIOTLB/PCIfront being at fault here.. Let me do that if your  2.6.33
>> > VM guest is running fine for the last two days.
>> 
>> 
>> 
>> 
>> -- 
>> Best regards,
>>  Sander                            mailto:linux@eikelenboom.it



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

[-- Attachment #2: interrupts.txt --]
[-- Type: text/plain, Size: 1249 bytes --]

            CPU0       
  86:          0  xen-pirq-pcifront-msi-x  xhci_hcd
  87:    4664997  xen-pirq-pcifront-msi-x  xhci_hcd
1268:       2585   xen-dyn-event     eth0
1269:       7108   xen-dyn-event     blkif
1270:       4491   xen-dyn-event     blkif
1271:         28   xen-dyn-event     blkif
1272:        775   xen-dyn-event     hvc_console
1273:          1   xen-dyn-event     pcifront
1274:        642   xen-dyn-event     xenbus
1275:          0   xen-dyn-ipi       callfuncsingle0
1276:          0   xen-dyn-virq      debug0
1277:          0   xen-dyn-ipi       callfunc0
1278:          0   xen-dyn-ipi       resched0
1279:     631711   xen-dyn-virq      timer0
 NMI:          0   Non-maskable interrupts
 LOC:          0   Local timer interrupts
 SPU:          0   Spurious interrupts
 PMI:          0   Performance monitoring interrupts
 PND:          0   Performance pending work
 RES:          0   Rescheduling interrupts
 CAL:          0   Function call interrupts
 TLB:          0   TLB shootdowns
 TRM:          0   Thermal event interrupts
 THR:          0   Threshold APIC interrupts
 MCE:          0   Machine check exceptions
 MCP:          0   Machine check polls
 ERR:          0
 MIS:          0

[-- Attachment #3: meminfo.txt --]
[-- Type: text/plain, Size: 1021 bytes --]

MemTotal:         440068 kB
MemFree:            9128 kB
Buffers:            9576 kB
Cached:           285616 kB
SwapCached:            0 kB
Active:            65168 kB
Inactive:         272952 kB
Active(anon):      20516 kB
Inactive(anon):    22628 kB
Active(file):      44652 kB
Inactive(file):   250324 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        524284 kB
SwapFree:         524284 kB
Dirty:                32 kB
Writeback:             0 kB
AnonPages:         42968 kB
Mapped:            11580 kB
Shmem:               216 kB
Slab:              80792 kB
SReclaimable:      16836 kB
SUnreclaim:        63956 kB
KernelStack:         520 kB
PageTables:         4168 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      744316 kB
Committed_AS:     182284 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        3556 kB
VmallocChunk:   34359734779 kB
DirectMap4k:      524288 kB
DirectMap2M:           0 kB

[-- Attachment #4: slabinfo.txt --]
[-- Type: text/plain, Size: 18549 bytes --]

slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
fuse_request           0      0    632    6    1 : tunables   54   27    0 : slabdata      0      0      0
fuse_inode             0      0    768    5    1 : tunables   54   27    0 : slabdata      0      0      0
bridge_fdb_cache       0      0     64   59    1 : tunables  120   60    0 : slabdata      0      0      0
flow_cache             0      0     96   40    1 : tunables  120   60    0 : slabdata      0      0      0
dm_raid1_read_record      0      0   1064    7    2 : tunables   24   12    0 : slabdata      0      0      0
dm_snap_tracked_chunk      0      0     24  144    1 : tunables  120   60    0 : slabdata      0      0      0
dm_snap_pending_exception      0      0     80   48    1 : tunables  120   60    0 : slabdata      0      0      0
dm_exception           0      0     32  112    1 : tunables  120   60    0 : slabdata      0      0      0
dm_mpath_io            0      0     16  202    1 : tunables  120   60    0 : slabdata      0      0      0
dm_crypt_io            0      0    152   25    1 : tunables  120   60    0 : slabdata      0      0      0
kcopyd_job             0      0    384   10    1 : tunables   54   27    0 : slabdata      0      0      0
io                     0      0     64   59    1 : tunables  120   60    0 : slabdata      0      0      0
dm_rq_clone_bio_info      0      0     16  202    1 : tunables  120   60    0 : slabdata      0      0      0
dm_rq_target_io        0      0    368   10    1 : tunables   54   27    0 : slabdata      0      0      0
dm_target_io           0      0     24  144    1 : tunables  120   60    0 : slabdata      0      0      0
dm_io                  0      0     40   92    1 : tunables  120   60    0 : slabdata      0      0      0
uhci_urb_priv          0      0     56   67    1 : tunables  120   60    0 : slabdata      0      0      0
cfq_io_context        17     30    128   30    1 : tunables  120   60    0 : slabdata      1      1      0
cfq_queue             18     34    232   17    1 : tunables  120   60    0 : slabdata      2      2      0
bsg_cmd                0      0    312   12    1 : tunables   54   27    0 : slabdata      0      0      0
mqueue_inode_cache      1      4    960    4    1 : tunables   54   27    0 : slabdata      1      1      0
extent_map             0      0     88   44    1 : tunables  120   60    0 : slabdata      0      0      0
extent_buffers         0      0    144   27    1 : tunables  120   60    0 : slabdata      0      0      0
extent_state           0      0    128   30    1 : tunables  120   60    0 : slabdata      0      0      0
btrfs_path_cache       0      0    144   27    1 : tunables  120   60    0 : slabdata      0      0      0
btrfs_transaction_cache      0      0    232   17    1 : tunables  120   60    0 : slabdata      0      0      0
btrfs_trans_handle_cache      0      0     64   59    1 : tunables  120   60    0 : slabdata      0      0      0
btrfs_inode_cache      0      0   1024    4    1 : tunables   54   27    0 : slabdata      0      0      0
nilfs2_btree_path_cache      0      0   1792    2    1 : tunables   24   12    0 : slabdata      0      0      0
nilfs2_segbuf_cache      0      0    232   17    1 : tunables  120   60    0 : slabdata      0      0      0
nilfs2_transaction_cache      0      0     40   92    1 : tunables  120   60    0 : slabdata      0      0      0
nilfs2_inode_cache      0      0    960    4    1 : tunables   54   27    0 : slabdata      0      0      0
jbd2_journal_handle      0      0     24  144    1 : tunables  120   60    0 : slabdata      0      0      0
jbd2_journal_head      0      0    112   34    1 : tunables  120   60    0 : slabdata      0      0      0
jbd2_revoke_table      0      0     16  202    1 : tunables  120   60    0 : slabdata      0      0      0
jbd2_revoke_record      0      0     32  112    1 : tunables  120   60    0 : slabdata      0      0      0
journal_handle         2    144     24  144    1 : tunables  120   60    0 : slabdata      1      1      0
journal_head          17     34    112   34    1 : tunables  120   60    0 : slabdata      1      1      0
revoke_table           4    202     16  202    1 : tunables  120   60    0 : slabdata      1      1      0
revoke_record          0      0     32  112    1 : tunables  120   60    0 : slabdata      0      0      0
ext4_inode_cache       0      0   1008    4    1 : tunables   54   27    0 : slabdata      0      0      0
ext4_xattr             0      0     88   44    1 : tunables  120   60    0 : slabdata      0      0      0
ext4_free_block_extents      0      0     56   67    1 : tunables  120   60    0 : slabdata      0      0      0
ext4_alloc_context      0      0    144   27    1 : tunables  120   60    0 : slabdata      0      0      0
ext4_prealloc_space      0      0    104   37    1 : tunables  120   60    0 : slabdata      0      0      0
ext4_system_zone       0      0     40   92    1 : tunables  120   60    0 : slabdata      0      0      0
ext2_inode_cache       0      0    824    4    1 : tunables   54   27    0 : slabdata      0      0      0
ext2_xattr             0      0     88   44    1 : tunables  120   60    0 : slabdata      0      0      0
ext3_inode_cache    3922   3940    848    4    1 : tunables   54   27    0 : slabdata    985    985      0
ext3_xattr             0      0     88   44    1 : tunables  120   60    0 : slabdata      0      0      0
dquot                  0      0    256   15    1 : tunables  120   60    0 : slabdata      0      0      0
kioctx                 0      0    384   10    1 : tunables   54   27    0 : slabdata      0      0      0
kiocb                  0      0    256   15    1 : tunables  120   60    0 : slabdata      0      0      0
inotify_event_private_data      0      0     32  112    1 : tunables  120   60    0 : slabdata      0      0      0
inotify_inode_mark_entry     23     34    112   34    1 : tunables  120   60    0 : slabdata      1      1      0
dnotify_mark_entry      0      0    112   34    1 : tunables  120   60    0 : slabdata      0      0      0
dnotify_struct         0      0     32  112    1 : tunables  120   60    0 : slabdata      0      0      0
fasync_cache           0      0     48   77    1 : tunables  120   60    0 : slabdata      0      0      0
pid_namespace          0      0   2112    3    2 : tunables   24   12    0 : slabdata      0      0      0
nsproxy                0      0     48   77    1 : tunables  120   60    0 : slabdata      0      0      0
posix_timers_cache      0      0    176   22    1 : tunables  120   60    0 : slabdata      0      0      0
uid_cache              4     30    128   30    1 : tunables  120   60    0 : slabdata      1      1      0
UNIX                  12     18    832    9    2 : tunables   54   27    0 : slabdata      2      2      0
ip_mrt_cache           0      0    128   30    1 : tunables  120   60    0 : slabdata      0      0      0
UDP-Lite               0      0    832    9    2 : tunables   54   27    0 : slabdata      0      0      0
tcp_bind_bucket        3    112     32  112    1 : tunables  120   60    0 : slabdata      1      1      0
inet_peer_cache        0      0     64   59    1 : tunables  120   60    0 : slabdata      0      0      0
secpath_cache          0      0     64   59    1 : tunables  120   60    0 : slabdata      0      0      0
xfrm_dst_cache         0      0    448    8    1 : tunables   54   27    0 : slabdata      0      0      0
ip_fib_alias           0      0     32  112    1 : tunables  120   60    0 : slabdata      0      0      0
ip_fib_hash            9     53     72   53    1 : tunables  120   60    0 : slabdata      1      1      0
ip_dst_cache          11     20    384   10    1 : tunables   54   27    0 : slabdata      2      2      0
arp_cache              2     15    256   15    1 : tunables  120   60    0 : slabdata      1      1      0
RAW                    2      9    832    9    2 : tunables   54   27    0 : slabdata      1      1      0
UDP                    0      0    832    9    2 : tunables   54   27    0 : slabdata      0      0      0
tw_sock_TCP            0      0    256   15    1 : tunables  120   60    0 : slabdata      0      0      0
request_sock_TCP       0      0    128   30    1 : tunables  120   60    0 : slabdata      0      0      0
TCP                    5      8   1664    4    2 : tunables   24   12    0 : slabdata      2      2      0
eventpoll_pwq          5     53     72   53    1 : tunables  120   60    0 : slabdata      1      1      0
eventpoll_epi          5     30    128   30    1 : tunables  120   60    0 : slabdata      1      1      0
sgpool-128             2      2   5120    1    2 : tunables    8    4    0 : slabdata      2      2      0
sgpool-64              2      3   2560    3    2 : tunables   24   12    0 : slabdata      1      1      0
sgpool-32              2      3   1280    3    1 : tunables   24   12    0 : slabdata      1      1      0
sgpool-16              2      6    640    6    1 : tunables   54   27    0 : slabdata      1      1      0
sgpool-8               2     12    320   12    1 : tunables   54   27    0 : slabdata      1      1      0
scsi_data_buffer       0      0     24  144    1 : tunables  120   60    0 : slabdata      0      0      0
blkdev_queue          19     21   2344    3    2 : tunables   24   12    0 : slabdata      7      7      0
blkdev_requests       12     12    328   12    1 : tunables   54   27    0 : slabdata      1      1      0
blkdev_ioc            16     59     64   59    1 : tunables  120   60    0 : slabdata      1      1      0
fsnotify_event_holder      0      0     24  144    1 : tunables  120   60    0 : slabdata      0      0      0
fsnotify_event         0      0    104   37    1 : tunables  120   60    0 : slabdata      0      0      0
bio-0                  2     20    192   20    1 : tunables  120   60    0 : slabdata      1      1      0
biovec-256             2      2   4096    1    1 : tunables   24   12    0 : slabdata      2      2      0
biovec-128             0      0   2048    2    1 : tunables   24   12    0 : slabdata      0      0      0
biovec-64              0      0   1024    4    1 : tunables   54   27    0 : slabdata      0      0      0
biovec-16              0      0    256   15    1 : tunables  120   60    0 : slabdata      0      0      0
sock_inode_cache      35     35    704    5    1 : tunables   54   27    0 : slabdata      7      7      0
skbuff_fclone_cache      7      7    512    7    1 : tunables   54   27    0 : slabdata      1      1      0
skbuff_head_cache     66    165    256   15    1 : tunables  120   60    0 : slabdata     11     11      0
file_lock_cache       22     22    176   22    1 : tunables  120   60    0 : slabdata      1      1      0
shmem_inode_cache    507    540    808    5    1 : tunables   54   27    0 : slabdata    108    108      0
task_delay_info       81    136    112   34    1 : tunables  120   60    0 : slabdata      4      4      0
taskstats              2     12    328   12    1 : tunables   54   27    0 : slabdata      1      1      0
proc_inode_cache     252    252    664    6    1 : tunables   54   27    0 : slabdata     42     42      0
sigqueue               0      0    160   24    1 : tunables  120   60    0 : slabdata      0      0      0
bdev_cache            20     20    896    4    1 : tunables   54   27    0 : slabdata      5      5      0
sysfs_dir_cache     3501   3504     80   48    1 : tunables  120   60    0 : slabdata     73     73      0
mnt_cache             22     30    256   15    1 : tunables  120   60    0 : slabdata      2      2      0
filp                 540    540    192   20    1 : tunables  120   60    0 : slabdata     27     27      0
inode_cache         1016   1050    616    6    1 : tunables   54   27    0 : slabdata    175    175      0
dentry              4121   5660    192   20    1 : tunables  120   60    0 : slabdata    283    283      0
names_cache            1      1   4096    1    1 : tunables   24   12    0 : slabdata      1      1      0
key_jar                0      0    192   20    1 : tunables  120   60    0 : slabdata      0      0      0
buffer_head        68770  72409    104   37    1 : tunables  120   60    0 : slabdata   1957   1957      0
vm_area_struct      3298   3358    168   23    1 : tunables  120   60    0 : slabdata    146    146      0
mm_struct             45     52    896    4    1 : tunables   54   27    0 : slabdata     13     13      0
fs_cache              30     59     64   59    1 : tunables  120   60    0 : slabdata      1      1      0
files_cache           32     66    704   11    2 : tunables   54   27    0 : slabdata      6      6      0
signal_cache          64     92   1024    4    1 : tunables   54   27    0 : slabdata     23     23      0
sighand_cache         72     72   2112    3    2 : tunables   24   12    0 : slabdata     24     24      0
task_xstate           48     64    512    8    1 : tunables   54   27    0 : slabdata      8      8      0
task_struct           77     96   1712    4    2 : tunables   24   12    0 : slabdata     24     24      0
cred_jar             160    160    192   20    1 : tunables  120   60    0 : slabdata      8      8      0
anon_vma_chain      3163   3542     48   77    1 : tunables  120   60    0 : slabdata     46     46      0
anon_vma            1694   2304     24  144    1 : tunables  120   60    0 : slabdata     16     16      0
pid                  120    120    128   30    1 : tunables  120   60    0 : slabdata      4      4      0
idr_layer_cache      156    161    544    7    1 : tunables   54   27    0 : slabdata     23     23      0
kmemleak_scan_area    422    448     32  112    1 : tunables  120   60    0 : slabdata      4      4      0
kmemleak_object   142008 159024    320   12    1 : tunables   54   27    0 : slabdata  13252  13252      0
radix_tree_node     5202   5264    552    7    1 : tunables   54   27    0 : slabdata    752    752      0
size-4194304(DMA)      0      0 4194304    1 1024 : tunables    1    1    0 : slabdata      0      0      0
size-4194304           0      0 4194304    1 1024 : tunables    1    1    0 : slabdata      0      0      0
size-2097152(DMA)      0      0 2097152    1  512 : tunables    1    1    0 : slabdata      0      0      0
size-2097152           0      0 2097152    1  512 : tunables    1    1    0 : slabdata      0      0      0
size-1048576(DMA)      0      0 1048576    1  256 : tunables    1    1    0 : slabdata      0      0      0
size-1048576           0      0 1048576    1  256 : tunables    1    1    0 : slabdata      0      0      0
size-524288(DMA)       0      0 524288    1  128 : tunables    1    1    0 : slabdata      0      0      0
size-524288            0      0 524288    1  128 : tunables    1    1    0 : slabdata      0      0      0
size-262144(DMA)       0      0 262144    1   64 : tunables    1    1    0 : slabdata      0      0      0
size-262144            0      0 262144    1   64 : tunables    1    1    0 : slabdata      0      0      0
size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-131072            0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-65536             0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-32768             1      1  32768    1    8 : tunables    8    4    0 : slabdata      1      1      0
size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
size-16384             4      4  16384    1    4 : tunables    8    4    0 : slabdata      4      4      0
size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0
size-8192              7      7   8192    1    2 : tunables    8    4    0 : slabdata      7      7      0
size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    0 : slabdata      0      0      0
size-4096             25     25   4096    1    1 : tunables   24   12    0 : slabdata     25     25      0
size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    0 : slabdata      0      0      0
size-2048             50     50   2048    2    1 : tunables   24   12    0 : slabdata     25     25      0
size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    0 : slabdata      0      0      0
size-1024            324    324   1024    4    1 : tunables   54   27    0 : slabdata     81     81      0
size-512(DMA)          0      0    512    8    1 : tunables   54   27    0 : slabdata      0      0      0
size-512             288    288    512    8    1 : tunables   54   27    0 : slabdata     36     36      0
size-256(DMA)          0      0    256   15    1 : tunables  120   60    0 : slabdata      0      0      0
size-256              66     75    256   15    1 : tunables  120   60    0 : slabdata      5      5      0
size-192(DMA)          0      0    192   20    1 : tunables  120   60    0 : slabdata      0      0      0
size-192           33260  33260    192   20    1 : tunables  120   60    0 : slabdata   1663   1663      0
size-128(DMA)          0      0    128   30    1 : tunables  120   60    0 : slabdata      0      0      0
size-64(DMA)           0      0     64   59    1 : tunables  120   60    0 : slabdata      0      0      0
size-64             5719   5959     64   59    1 : tunables  120   60    0 : slabdata    101    101      0
size-32(DMA)           0      0     32  112    1 : tunables  120   60    0 : slabdata      0      0      0
size-128             690    690    128   30    1 : tunables  120   60    0 : slabdata     23     23      0
size-32             4704   4704     32  112    1 : tunables  120   60    0 : slabdata     42     42      0
kmem_cache           168    170    384   10    1 : tunables   54   27    0 : slabdata     17     17      0

[-- Attachment #5: vmallacinfo.txt --]
[-- Type: text/plain, Size: 2329 bytes --]

0xffffc90000000000-0xffffc90000021000  135168 arch_gnttab_map_shared+0x24/0x56 ioremap
0xffffc90000022000-0xffffc90000024000    8192 init_vdso_vars+0xef/0x221
0xffffc90000026000-0xffffc90000028000    8192 pci_enable_msix+0x1dc/0x3eb phys=fe401000 ioremap
0xffffc90000028000-0xffffc9000002b000   12288 usb_hcd_pci_probe+0x15f/0x30f phys=fe400000 ioremap
0xffffc9000002c000-0xffffc9000004d000  135168 sys_swapon+0x4e7/0xae2 pages=32 vmalloc
0xffffc9000004e000-0xffffc9000011a000  835584 __videobuf_mmap_mapper+0x116/0x200 [videobuf_vmalloc] pages=203 vmalloc user
0xffffc9000011b000-0xffffc900001e7000  835584 __videobuf_mmap_mapper+0x116/0x200 [videobuf_vmalloc] pages=203 vmalloc user
0xffffc900001e8000-0xffffc900002b4000  835584 __videobuf_mmap_mapper+0x116/0x200 [videobuf_vmalloc] pages=203 vmalloc user
0xffffc900002b5000-0xffffc90000381000  835584 __videobuf_mmap_mapper+0x116/0x200 [videobuf_vmalloc] pages=203 vmalloc user
0xffffffffa0000000-0xffffffffa0004000   16384 module_alloc_update_bounds+0xe/0x5a pages=3 vmalloc
0xffffffffa0005000-0xffffffffa0007000    8192 module_alloc_update_bounds+0xe/0x5a pages=1 vmalloc
0xffffffffa0008000-0xffffffffa000d000   20480 module_alloc_update_bounds+0xe/0x5a pages=4 vmalloc
0xffffffffa000e000-0xffffffffa0011000   12288 module_alloc_update_bounds+0xe/0x5a pages=2 vmalloc
0xffffffffa0012000-0xffffffffa0016000   16384 module_alloc_update_bounds+0xe/0x5a pages=3 vmalloc
0xffffffffa0017000-0xffffffffa001c000   20480 module_alloc_update_bounds+0xe/0x5a pages=4 vmalloc
0xffffffffa001d000-0xffffffffa001f000    8192 module_alloc_update_bounds+0xe/0x5a pages=1 vmalloc
0xffffffffa0020000-0xffffffffa0024000   16384 module_alloc_update_bounds+0xe/0x5a pages=3 vmalloc
0xffffffffa0025000-0xffffffffa0029000   16384 module_alloc_update_bounds+0xe/0x5a pages=3 vmalloc
0xffffffffa002a000-0xffffffffa0035000   45056 module_alloc_update_bounds+0xe/0x5a pages=10 vmalloc
0xffffffffa0036000-0xffffffffa003b000   20480 module_alloc_update_bounds+0xe/0x5a pages=4 vmalloc
0xffffffffa003c000-0xffffffffa0050000   81920 module_alloc_update_bounds+0xe/0x5a pages=19 vmalloc
0xffffffffa0059000-0xffffffffa005d000   16384 module_alloc_update_bounds+0xe/0x5a pages=3 vmalloc
0xffffffffa005e000-0xffffffffa006c000   57344 module_alloc_update_bounds+0xe/0x5a pages=13 vmalloc

[-- Attachment #6: vmstat.txt --]
[-- Type: text/plain, Size: 1446 bytes --]

nr_free_pages 2253
nr_inactive_anon 5657
nr_active_anon 5129
nr_inactive_file 62586
nr_active_file 11165
nr_unevictable 0
nr_mlock 0
nr_anon_pages 10742
nr_mapped 2895
nr_file_pages 73805
nr_dirty 4
nr_writeback 0
nr_slab_reclaimable 4209
nr_slab_unreclaimable 15988
nr_page_table_pages 1042
nr_kernel_stack 64
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 54
pgpgin 68514
pgpgout 1132904
pswpin 0
pswpout 0
pgalloc_dma 7283
pgalloc_dma32 513627
pgalloc_normal 0
pgalloc_movable 0
pgfree 523304
pgactivate 14814
pgdeactivate 5664
pgfault 1005226
pgmajfault 575
pgrefill_dma 32
pgrefill_dma32 5632
pgrefill_normal 0
pgrefill_movable 0
pgsteal_dma 0
pgsteal_dma32 59063
pgsteal_normal 0
pgsteal_movable 0
pgscan_kswapd_dma 0
pgscan_kswapd_dma32 60864
pgscan_kswapd_normal 0
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 0
pgscan_direct_normal 0
pgscan_direct_movable 0
pginodesteal 0
slabs_scanned 12544
kswapd_steal 59063
kswapd_inodesteal 18737
kswapd_low_wmark_hit_quickly 9
kswapd_high_wmark_hit_quickly 33
kswapd_skip_congestion_wait 0
pageoutrun 1158
allocstall 0
pgrotated 0
unevictable_pgs_culled 0
unevictable_pgs_scanned 0
unevictable_pgs_rescued 0
unevictable_pgs_mlocked 0
unevictable_pgs_munlocked 0
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0

[-- Attachment #7: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel