[ kvm-Bugs-1881532 ] Network access seg faults KVM on large-memory machine

[ kvm-Bugs-1881532 ] Network access seg faults KVM on large-memory machine

[SourceForge.net]

Bugs item #1881532, was opened at 2008-01-28 18:16
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1881532&group_id=180599

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Scott Pakin (pakin)
Assigned to: Nobody/Anonymous (nobody)
Summary: Network access seg faults KVM on large-memory machine

Initial Comment:
The TCP-handling code in qemu/slirp is not 64-bit clean.  On a system
with tons of memory -- I'm running on a system with just under 48GB of
RAM -- KVM dies with a segmentation fault when I try to access the
network.

Specifically, I boot the Ubuntu live CD image, open a terminal, and
try to ssh to the host machine.  KVM segfaults before ssh returns.

For completeness, here are the parameters requested on the "Submitting
a bug report" page:

    CPU model:     Dual-Core AMD Opteron(tm) Processor 8212
    KVM version:   60
    Host kernel:   Linux version 2.6.18-8.1.14.el5 (mockbuild-Bqj+TZ04CV0YyQU+NhOpU0B+6BGkLq7r@public.gmane.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Thu Sep 27 19:05:32 EDT 2007
    Guest arch:    x86_64
    Guest OS:      Ubuntu Linux 7.10
    Guest bitness: 64
    Guest kernel:  Linux version 2.6.22-14-generic (buildd@crested) (gcc version 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)) #1 SMP Sun Oct 14 21:45:15 GMT 2007
    Command line:  qemu-system-x86_64 -cdrom ubuntu-7.10-desktop-amd64.iso -boot d -localtime -m 1024
    With -no-kvm:  [still fails]

Fortunately, I managed to track down the problem.  First, note how the
seg_next field is declared in a TCP control block:

    qemu/slirp/tcp_var.h:
      43 #if SIZEOF_CHAR_P == 4
      44  typedef struct tcpiphdr *tcpiphdrp_32;
      45 #else
      46  typedef u_int32_t tcpiphdrp_32;
      47 #endif
      ...
      52 struct tcpcb {
      53         tcpiphdrp_32 seg_next;  /* sequencing queue */
      54         tcpiphdrp_32 seg_prev;

Because this is a 64-bit build, SIZEOF_CHAR_P is 8 so tcpiphdrp_32 --
and hence seg_next -- is defined as a u_int32_t, a 32-bit integer.

Second, note how tp->seg_next is defined:

    qemu/slirp/tcp_subr.c:
      189 tcp_newtcpcb(so)
      190         struct socket *so;
      191 {
      192         register struct tcpcb *tp;
      193
      194         tp = (struct tcpcb *)malloc(sizeof(*tp));
      195         if (tp == NULL)
      196                 return ((struct tcpcb *)0);
      197
      198         memset((char *) tp, 0, sizeof(struct tcpcb));
      199         tp->seg_next = tp->seg_prev = (tcpiphdrp_32)tp;

It's taking a 64-bit pointer, tp, and assigning it to a seg_next
field, which is a 32-bit integer.  Not surprisingly, a SIGSEGV is
generated when that 32-bit integer is cast back to a 64-bit pointer
and dereferenced:

    qemu/slirp/tcp_input.c:
      207         ti = (struct tcpiphdr *) tp->seg_next;
      208         if (ti == (struct tcpiphdr *)tp || ti->ti_seq != tp->rcv_nxt)
      209                 return (0);

I believe the following patch should fix the problem:

===================== BEGIN kvm_tcp_seg_next.patch =====================
--- qemu/slirp/tcp_var.h.ORIG   2008-01-28 17:27:09.000000000 -0700
+++ qemu/slirp/tcp_var.h        2008-01-28 17:27:20.000000000 -0700
@@ -40,11 +40,7 @@
 #include "tcpip.h"
 #include "tcp_timer.h"

-#if SIZEOF_CHAR_P == 4
- typedef struct tcpiphdr *tcpiphdrp_32;
-#else
- typedef u_int32_t tcpiphdrp_32;
-#endif
+typedef struct tcpiphdr *tcpiphdrp_32;

 /*
  * Tcp control block, one per tcp; fields:
====================== END kvm_tcp_seg_next.patch ======================

As far as I can tell, tcpiphdr_32 is used only to define seg_next and
seg_prev so it _should_ be safe to define it as an ordinary pointer
regardless of 32/64-bitness.

There are a bunch of other potential sources of similar problems in
qemu/slirp, but I haven't yet triggered these:

    slirp/ip_input.c:278: warning: cast from pointer to integer of different size
    slirp/ip_input.c:298: warning: cast from pointer to integer of different size
    slirp/ip_input.c:427: warning: cast from pointer to integer of different size
    slirp/ip_input.c:429: warning: cast from pointer to integer of different size
    slirp/ip_input.c:430: warning: cast from pointer to integer of different size
    slirp/misc.c:115: warning: cast from pointer to integer of different size
    slirp/misc.c:116: warning: cast from pointer to integer of different size
    slirp/misc.c:118: warning: cast from pointer to integer of different size
    slirp/tcp_input.c:173: warning: cast from pointer to integer of different size

Still, you might want to have a look at those, too, to make sure that
KVM will work on large-memory machines.



----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1881532&group_id=180599

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[ kvm-Bugs-1881532 ] Network access seg faults KVM on large-memory machine

[SourceForge.net]

Bugs item #1881532, was opened at 2008-01-29 02:16
Message generated for change (Comment added) made by ravpl
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1881532&group_id=180599

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Scott Pakin (pakin)
Assigned to: Nobody/Anonymous (nobody)
Summary: Network access seg faults KVM on large-memory machine

Initial Comment:
The TCP-handling code in qemu/slirp is not 64-bit clean.  On a system
with tons of memory -- I'm running on a system with just under 48GB of
RAM -- KVM dies with a segmentation fault when I try to access the
network.

Specifically, I boot the Ubuntu live CD image, open a terminal, and
try to ssh to the host machine.  KVM segfaults before ssh returns.

For completeness, here are the parameters requested on the "Submitting
a bug report" page:

    CPU model:     Dual-Core AMD Opteron(tm) Processor 8212
    KVM version:   60
    Host kernel:   Linux version 2.6.18-8.1.14.el5 (mockbuild@builder6.centos.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Thu Sep 27 19:05:32 EDT 2007
    Guest arch:    x86_64
    Guest OS:      Ubuntu Linux 7.10
    Guest bitness: 64
    Guest kernel:  Linux version 2.6.22-14-generic (buildd@crested) (gcc version 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)) #1 SMP Sun Oct 14 21:45:15 GMT 2007
    Command line:  qemu-system-x86_64 -cdrom ubuntu-7.10-desktop-amd64.iso -boot d -localtime -m 1024
    With -no-kvm:  [still fails]

Fortunately, I managed to track down the problem.  First, note how the
seg_next field is declared in a TCP control block:

    qemu/slirp/tcp_var.h:
      43 #if SIZEOF_CHAR_P == 4
      44  typedef struct tcpiphdr *tcpiphdrp_32;
      45 #else
      46  typedef u_int32_t tcpiphdrp_32;
      47 #endif
      ...
      52 struct tcpcb {
      53         tcpiphdrp_32 seg_next;  /* sequencing queue */
      54         tcpiphdrp_32 seg_prev;

Because this is a 64-bit build, SIZEOF_CHAR_P is 8 so tcpiphdrp_32 --
and hence seg_next -- is defined as a u_int32_t, a 32-bit integer.

Second, note how tp->seg_next is defined:

    qemu/slirp/tcp_subr.c:
      189 tcp_newtcpcb(so)
      190         struct socket *so;
      191 {
      192         register struct tcpcb *tp;
      193
      194         tp = (struct tcpcb *)malloc(sizeof(*tp));
      195         if (tp == NULL)
      196                 return ((struct tcpcb *)0);
      197
      198         memset((char *) tp, 0, sizeof(struct tcpcb));
      199         tp->seg_next = tp->seg_prev = (tcpiphdrp_32)tp;

It's taking a 64-bit pointer, tp, and assigning it to a seg_next
field, which is a 32-bit integer.  Not surprisingly, a SIGSEGV is
generated when that 32-bit integer is cast back to a 64-bit pointer
and dereferenced:

    qemu/slirp/tcp_input.c:
      207         ti = (struct tcpiphdr *) tp->seg_next;
      208         if (ti == (struct tcpiphdr *)tp || ti->ti_seq != tp->rcv_nxt)
      209                 return (0);

I believe the following patch should fix the problem:

===================== BEGIN kvm_tcp_seg_next.patch =====================
--- qemu/slirp/tcp_var.h.ORIG   2008-01-28 17:27:09.000000000 -0700
+++ qemu/slirp/tcp_var.h        2008-01-28 17:27:20.000000000 -0700
@@ -40,11 +40,7 @@
 #include "tcpip.h"
 #include "tcp_timer.h"

-#if SIZEOF_CHAR_P == 4
- typedef struct tcpiphdr *tcpiphdrp_32;
-#else
- typedef u_int32_t tcpiphdrp_32;
-#endif
+typedef struct tcpiphdr *tcpiphdrp_32;

 /*
  * Tcp control block, one per tcp; fields:
====================== END kvm_tcp_seg_next.patch ======================

As far as I can tell, tcpiphdr_32 is used only to define seg_next and
seg_prev so it _should_ be safe to define it as an ordinary pointer
regardless of 32/64-bitness.

There are a bunch of other potential sources of similar problems in
qemu/slirp, but I haven't yet triggered these:

    slirp/ip_input.c:278: warning: cast from pointer to integer of different size
    slirp/ip_input.c:298: warning: cast from pointer to integer of different size
    slirp/ip_input.c:427: warning: cast from pointer to integer of different size
    slirp/ip_input.c:429: warning: cast from pointer to integer of different size
    slirp/ip_input.c:430: warning: cast from pointer to integer of different size
    slirp/misc.c:115: warning: cast from pointer to integer of different size
    slirp/misc.c:116: warning: cast from pointer to integer of different size
    slirp/misc.c:118: warning: cast from pointer to integer of different size
    slirp/tcp_input.c:173: warning: cast from pointer to integer of different size

Still, you might want to have a look at those, too, to make sure that
KVM will work on large-memory machines.



----------------------------------------------------------------------

Comment By: Rafal Wijata (ravpl)
Date: 2008-07-07 12:28

Message:
Logged In: YES 
user_id=996150
Originator: NO

I can confirm, that kvm-70 still have this issue(or similar). I tried
8139cp & e1000 network drivers, both segfaults with following bt.
host is Fedora8 64bit 2.6.25.6-27.fc8 with kvm modules loaded from kvm-70
guest is running RHEL51 64bit (2.6.18-92.1.6.el5) 6Cpus, 5G RAM

Note, that I had no such issue, when guest was RHEL4u6 64bit. Just
upgraded and encountered this problem.

#0  tb_invalidate_phys_page_range (start=5359317010, end=5359317195,
is_cpu_write_access=0) at /usr/src/kvm-70/qemu/exec.c:791
791         if (!p->code_bitmap &&
(gdb) bt
#0  tb_invalidate_phys_page_range (start=5359317010, end=5359317195,
is_cpu_write_access=0) at /usr/src/kvm-70/qemu/exec.c:791
#1  0x0000000000494e98 in cpu_physical_memory_rw (addr=5896187922,
buf=0x28e55cc "", len=185, is_write=1) at
/usr/src/kvm-70/qemu/exec.c:2825
#2  0x0000000000429b7e in e1000_receive (opaque=0x396b990, buf=0x28e55cc
"", size=<value optimized out>) at ../cpu-all.h:914
#3  0x000000000040603b in qemu_send_packet (vc1=0x28e65e0, buf=0x28e55cc
"", size=181) at /usr/src/kvm-70/qemu/vl.c:3820
#4  0x0000000000406c3c in tap_send (opaque=<value optimized out>) at
/usr/src/kvm-70/qemu/vl.c:4162
#5  0x0000000000409e34 in main_loop_wait (timeout=<value optimized out>)
at /usr/src/kvm-70/qemu/vl.c:7523
#6  0x00000000004e6e02 in kvm_main_loop () at
/usr/src/kvm-70/qemu/qemu-kvm.c:587
#7  0x00000000004115f3 in main (argc=11, argv=0x7fffb6779ea8) at
/usr/src/kvm-70/qemu/vl.c:7587


----------------------------------------------------------------------

Comment By: Scott Pakin (pakin)
Date: 2008-01-30 21:31

Message:
Logged In: YES 
user_id=406633
Originator: YES

Done.  In a follow-up I also submitted an extended patch that addresses
the remaining pointer/integer casting warnings (http://tinyurl.com/24tel5).

----------------------------------------------------------------------

Comment By: Avi Kivity (avik)
Date: 2008-01-29 09:23

Message:
Logged In: YES 
user_id=539971
Originator: NO

I believe your patch is correct.  Can you send it to
kvm-devel@lists.sourceforge.net, and to qemu-devel as well?

Thanks.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1881532&group_id=180599