Host dies on NFS write

Host dies on NFS write

[Anders]

Hi,

I am running 32 bit Linux (Ubuntu Gutsy: 2.6.22-14-server) on host and
guest. Many kvm-intel versions, including kvm-58.

The guest is an NFS server. Whenever I write a large file from the
host, the guest freezes within a few seconds, and the host slowly
spirals into death as well. I am then unable to do most things, but
usually can do ctrl-alt-f1, ctrl-alt-delete to get a proper shutdown
of the host.

There is no Oops or other messages on either host or guest console.

When reading from the NFS, I do not see a problem. Also, a scp write
of the same file completes.

Running strace on the qemu host process shows that it stops in a
write() system call that never returns. The fd is the NFS drive, which
is "-hdb /dev/md1" on the host.

I have tried changing any option that I can think of (including using
all the nic drivers), both in qemu/kvm options and in the guest
kernel, but the problem persists. The only thing that helped was
-no-kvm.

Any pointers on how to debug this further? Thanks!


   Anders

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Host dies on NFS write

[Izik Eidus]

Anders wrote:
> Hi,
>
> I am running 32 bit Linux (Ubuntu Gutsy: 2.6.22-14-server) on host and
> guest. Many kvm-intel versions, including kvm-58.
>
> The guest is an NFS server. Whenever I write a large file from the
> host, the guest freezes within a few seconds, 
i dont know if it will help you, but it solved many ppl problems with 
network
try to run the vm with the rtl8139 network card

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Host dies on NFS write

[Yves Dorfsman]

On Sat, 29 Dec 2007, Izik Eidus wrote:

>> The guest is an NFS server. Whenever I write a large file from the
>> host, the guest freezes within a few seconds,
> i dont know if it will help you, but it solved many ppl problems with
> network
> try to run the vm with the rtl8139 network card

But using the 8139cp driver in Linux, not the 8139too. Solved my problem.

Yves.
----
Yves Dorfsman                                             yves-Qq95MN99UUsAvxtiuMwx3w@public.gmane.org
                                                    http://www.SollerS.ca


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Host dies on NFS write

[Anders]

Izik Eidus <izike-atKUWr5tajBWk0Htik3J/w@public.gmane.org> writes:

> Anders wrote:

[...]

>> The guest is an NFS server. Whenever I write a large file from the
>> host, the guest freezes within a few seconds,
>
> i dont know if it will help you, but it solved many ppl problems with
> network
> try to run the vm with the rtl8139 network card

The nic is already rtl8139 (also tried ne2k), so this is not the
solution. I tried 8139too and 8139cp drivers in guest, same thing.

Not sure if this is of any help, but here is the alt-sysrq-t output
when the host stalls:


 qemu-system-x D EF4C5CA4     0  6182      1 (NOTLB)
        ef4c5cb8 00000082 00000002 ef4c5ca4 ef4c5ca0 00000000 c131b0c0 3a71f261 
        c131ae20 c131b0e0 f0b0e92f 0000006a 0000069a c131ac60 dfcdcb70 c3615b00 
        00000000 0385e9bb 00000286 f70c1380 ef4c5cc8 ef4c5cc8 c0478f00 00000286 
 Call Trace:
  [__mod_timer+159/176] __mod_timer+0x9f/0xb0
  [schedule_timeout+74/192] schedule_timeout+0x4a/0xc0
  [process_timeout+0/16] process_timeout+0x0/0x10
  [schedule_timeout+69/192] schedule_timeout+0x45/0xc0
  [io_schedule_timeout+30/48] io_schedule_timeout+0x1e/0x30
  [congestion_wait+119/160] congestion_wait+0x77/0xa0
  [autoremove_wake_function+0/80] autoremove_wake_function+0x0/0x50
  [balance_dirty_pages_ratelimited_nr+272/464] balance_dirty_pages_ratelimited_nr+0x110/0x1d0
  [generic_file_buffered_write+769/1664] generic_file_buffered_write+0x301/0x680
  [<f8907620>] scsi_done+0x0/0x20 [scsi_mod]
  [<f8907d32>] scsi_dispatch_cmd+0x172/0x2c0 [scsi_mod]
  [current_fs_time+65/80] current_fs_time+0x41/0x50
  [__generic_file_aio_write_nolock+663/1360] __generic_file_aio_write_nolock+0x297/0x550
  [generic_file_aio_write_nolock+71/176] generic_file_aio_write_nolock+0x47/0xb0
  [<f89e27d1>] sd_rw_intr+0x81/0x3d0 [sd_mod]
  [do_sync_write+213/288] do_sync_write+0xd5/0x120
  [autoremove_wake_function+0/80] autoremove_wake_function+0x0/0x50
  [lapic_next_event+12/16] lapic_next_event+0xc/0x10
  [clockevents_program_event+132/256] clockevents_program_event+0x84/0x100
  [tick_program_event+68/112] tick_program_event+0x44/0x70
  [<f8f66800>] kvm_vcpu_ioctl+0x0/0x2f0 [kvm]
  [do_ioctl+43/192] do_ioctl+0x2b/0xc0
  [vfs_write+190/352] vfs_write+0xbe/0x160
  [vfs_llseek+60/80] vfs_llseek+0x3c/0x50
  [do_sync_write+0/288] do_sync_write+0x0/0x120
  [sys_write+65/112] sys_write+0x41/0x70
  [sysenter_past_esp+107/161] sysenter_past_esp+0x6b/0xa1


Regards,
Anders

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Host dies on NFS write

[Anders]

Anders <mail-e0wP/LNAh+3kORH8mtz0xlpr/1R2p/CL@public.gmane.org> writes:

[...]

> Not sure if this is of any help, but here is the alt-sysrq-t output
> when the host stalls:

[...]

>   [congestion_wait+119/160] congestion_wait+0x77/0xa0
>   [autoremove_wake_function+0/80] autoremove_wake_function+0x0/0x50
>   [balance_dirty_pages_ratelimited_nr+272/464] balance_dirty_pages_ratelimited_nr+0x110/0x1d0

Hi again.

Somehow, posting for help always seems to make you find the answer
yourself. It appears that I have hit a non-kvm problem with the 2.6.22
host kernel, as discussed in this thread:

        Processes spinning forever, apparently in lock_timer_base()?

        http://lkml.org/lkml/2007/8/1/469

Updating to the 2.6.24 kernel makes my NFS work properly
(unfortunately, other things start breaking).


Cheers,
Anders.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/